[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:18 dist-test-slave-gkw7 krb5kdc[3964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:18 dist-test-slave-gkw7 krb5kdc[3964](info): set up 2 sockets
Aug 09 01:52:18 dist-test-slave-gkw7 krb5kdc[3964](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 01:52:19.473963 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:19 dist-test-slave-gkw7 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:19.610350 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:33515
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:33515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:19.737952 3984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:19.738265 3984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:19.738350 3984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:19.742723 3984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:19.742812 3984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:19.742851 3984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:19.742905 3984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:19.742946 3984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:19.748487 3984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:33515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:19.750041 3984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:19.751422 3984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:19.764860 3990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:19.765018 3984 server_base.cc:1034] running on GCE node
W20260809 01:52:19.764920 3994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:19.764909 3991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:19.765892 3984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:19.767598 3984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:19.768810 3984 hybrid_clock.cc:648] HybridClock initialized: now 1786240339768765 us; error 83 us; skew 500 ppm
Aug 09 01:52:19 dist-test-slave-gkw7 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:19.772017 3984 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:19.773241 3984 webserver.cc:460] Webserver started at http://127.3.220.62:38757/ using document root <none> and password file <none>
I20260809 01:52:19.774247 3984 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:19.774365 3984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:19.774783 3984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:19.777385 3984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "97396031c0764f64becba8ece6e79013"
format_stamp: "Formatted at 2026-08-09 01:52:19 on dist-test-slave-gkw7"
server_key: "75a8995a567c0f99beaaf3c6c4b0eb01"
server_key_iv: "c5f3600f8789e4ff4faca0930afcc777"
server_key_version: "encryptionkey@0"
I20260809 01:52:19.778033 3984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "97396031c0764f64becba8ece6e79013"
format_stamp: "Formatted at 2026-08-09 01:52:19 on dist-test-slave-gkw7"
server_key: "75a8995a567c0f99beaaf3c6c4b0eb01"
server_key_iv: "c5f3600f8789e4ff4faca0930afcc777"
server_key_version: "encryptionkey@0"
I20260809 01:52:19.783341 3984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 01:52:19.786923 4006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:19.788516 3984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:52:19.788707 3984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "97396031c0764f64becba8ece6e79013"
format_stamp: "Formatted at 2026-08-09 01:52:19 on dist-test-slave-gkw7"
server_key: "75a8995a567c0f99beaaf3c6c4b0eb01"
server_key_iv: "c5f3600f8789e4ff4faca0930afcc777"
server_key_version: "encryptionkey@0"
I20260809 01:52:19.788822 3984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:19.811847 3984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:19.815503 3984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:19.815754 3984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:19.825088 3984 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:33515
I20260809 01:52:19.825133 4079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:33515 every 8 connection(s)
I20260809 01:52:19.826507 3984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:19.828559 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 3984
I20260809 01:52:19.828723 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:19.830216 3952 external_mini_cluster.cc:1428] Setting key 5f82b3707c5625b39480d9ecee9ac12b
I20260809 01:52:19.831619 4081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:19.842170 4081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013: Bootstrap starting.
I20260809 01:52:19.846855 4081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:19.848014 4081 log.cc:824] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:19.851305 4081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013: No bootstrap required, opened a new log
Aug 09 01:52:19 dist-test-slave-gkw7 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:19.857197 4081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97396031c0764f64becba8ece6e79013" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33515 } }
I20260809 01:52:19.857610 4081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:19.857733 4081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97396031c0764f64becba8ece6e79013, State: Initialized, Role: FOLLOWER
I20260809 01:52:19.858597 4081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97396031c0764f64becba8ece6e79013" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33515 } }
I20260809 01:52:19.858824 4081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:19.858961 4081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:19.859109 4081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:19.860376 4081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97396031c0764f64becba8ece6e79013" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33515 } }
I20260809 01:52:19.860464 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:19.834827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.3.220.62:33515)
0809 01:52:19.835338 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:19.835360 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:19.836694 (+ 1334us) server_negotiation.cc:407] Connection header received
0809 01:52:19.841298 (+ 4604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:19.841334 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:19.841731 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:19.842009 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:19.845882 (+ 3873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:19.846934 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:19.848168 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:19.848698 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:19.848868 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:19.854012 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:19.854060 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:19.854080 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:19.854115 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:19.856238 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:19.856808 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:19.856814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:19.856820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:19.856985 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:19.857321 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:19.857324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:19.857326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:19.858029 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:19.858352 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:19.859036 (+ 684us) server_negotiation.cc:299] Negotiation successful
0809 01:52:19.859377 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":157,"threads_started":1}
I20260809 01:52:19.861003 4081 leader_election.cc:302] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97396031c0764f64becba8ece6e79013; no voters:
I20260809 01:52:19.861389 4081 leader_election.cc:288] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:19.861500 4086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:19.861742 4086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [term 1 LEADER]: Becoming Leader. State: Replica: 97396031c0764f64becba8ece6e79013, State: Running, Role: LEADER
I20260809 01:52:19.862187 4086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97396031c0764f64becba8ece6e79013" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33515 } }
I20260809 01:52:19.863368 4081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:19.865180 4089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97396031c0764f64becba8ece6e79013. Latest consensus state: current_term: 1 leader_uuid: "97396031c0764f64becba8ece6e79013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97396031c0764f64becba8ece6e79013" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33515 } } }
I20260809 01:52:19.865315 4089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:19.865708 4091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97396031c0764f64becba8ece6e79013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97396031c0764f64becba8ece6e79013" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33515 } } }
I20260809 01:52:19.865831 4091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:19.866140 4101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:19.870934 4101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:19.878441 4101 catalog_manager.cc:1347] Generated new cluster ID: fb522f1480404134845efbb280471cdc
I20260809 01:52:19.878515 4101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:19.897333 4101 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:19.898376 4101 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:19.905510 4101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97396031c0764f64becba8ece6e79013: Generated new TSK 0
I20260809 01:52:19.906378 4101 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:19.944235 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:33515
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:20.069344 4117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:20.069672 4117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:20.069742 4117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:20.073868 4117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:20.073933 4117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:20.074009 4117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:20.079408 4117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:33515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:20.080663 4117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:20.081768 4117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:20.092157 4125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:20.092238 4126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:20.092263 4128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:20.092387 4117 server_base.cc:1034] running on GCE node
I20260809 01:52:20.092958 4117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:20.093760 4117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:20.094957 4117 hybrid_clock.cc:648] HybridClock initialized: now 1786240340094921 us; error 46 us; skew 500 ppm
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:20.098172 4117 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:20.099025 4117 webserver.cc:460] Webserver started at http://127.3.220.1:42811/ using document root <none> and password file <none>
I20260809 01:52:20.099558 4117 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:20.099617 4117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:20.099793 4117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:20.101933 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "59f6041ef14449348106151e95ef4ff2"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "8aa67a89d8616835f7624883d2615be4"
server_key_iv: "43469b36be13de203367833840d3cdc7"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.102456 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "59f6041ef14449348106151e95ef4ff2"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "8aa67a89d8616835f7624883d2615be4"
server_key_iv: "43469b36be13de203367833840d3cdc7"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.106132 4117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:20.108434 4139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.109730 4117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:20.109876 4117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "59f6041ef14449348106151e95ef4ff2"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "8aa67a89d8616835f7624883d2615be4"
server_key_iv: "43469b36be13de203367833840d3cdc7"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.109992 4117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:20.129808 4117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:20.132735 4117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:20.132893 4117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:20.133561 4117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:20.134614 4117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:20.134672 4117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.134716 4117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:20.134738 4117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.143894 4117 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:44991
I20260809 01:52:20.143918 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:44991 every 8 connection(s)
I20260809 01:52:20.144838 4117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:20.151705 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4117
I20260809 01:52:20.151816 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:20.152061 3952 external_mini_cluster.cc:1428] Setting key a08c50a3f24b421fdd4862a9f84b71ce
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:20.160476 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:20.147098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:47489 (local address 127.3.220.62:33515)
0809 01:52:20.147238 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:20.147243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:20.148149 (+ 906us) server_negotiation.cc:407] Connection header received
0809 01:52:20.150020 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:20.150023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:20.150069 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:20.150148 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:20.151388 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.152210 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.153386 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.153696 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.153810 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:20.156349 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:20.156355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:20.156359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:20.156397 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:20.158221 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.158725 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.158740 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.158744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.158849 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.159172 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.159178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.159183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.159450 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:20.159571 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:20.160045 (+ 474us) server_negotiation.cc:299] Negotiation successful
0809 01:52:20.160222 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 01:52:20.161229 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:20.147473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:33515 (local address 127.3.220.1:47489)
0809 01:52:20.147969 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:20.148026 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:20.149823 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:20.150354 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:20.150361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:20.150754 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:20.151249 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:20.151263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.152359 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.152362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:20.153248 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:20.153258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.153850 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.153855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:20.154019 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:20.154623 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:20.154640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:20.156182 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:20.158333 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:20.158338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:20.158350 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:20.158623 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:20.158944 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:20.158947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:20.158948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:20.159078 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:20.159558 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:20.159562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:20.159799 (+ 237us) client_negotiation.cc:769] Sending connection context
0809 01:52:20.159956 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 01:52:20.160261 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260809 01:52:20.162536 4309 heartbeater.cc:347] Connected to a master server at 127.3.220.62:33515
I20260809 01:52:20.162847 4309 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:52:20.163441 4309 heartbeater.cc:511] Master 127.3.220.62:33515 requested a full tablet report, sending...
I20260809 01:52:20.165591 4026 ts_manager.cc:194] Registered new tserver with Master: 59f6041ef14449348106151e95ef4ff2 (127.3.220.1:44991)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:52:20.167166 4026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:47489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:20.180625 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:33515
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:20.293268 4316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:20.293563 4316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:20.293642 4316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:20.297672 4316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:20.297734 4316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:20.297828 4316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:20.303042 4316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:33515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:20.304072 4316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:20.305078 4316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:20.314533 4328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:20.314541 4326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:20.314677 4325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:20.314980 4316 server_base.cc:1034] running on GCE node
I20260809 01:52:20.315388 4316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:20.315985 4316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:20.317147 4316 hybrid_clock.cc:648] HybridClock initialized: now 1786240340317141 us; error 46 us; skew 500 ppm
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:20.319847 4316 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:20.320710 4316 webserver.cc:460] Webserver started at http://127.3.220.2:33491/ using document root <none> and password file <none>
I20260809 01:52:20.321276 4316 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:20.321367 4316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:20.321647 4316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:20.323748 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "c450a736c36d4d82b267866bfa8cfce4"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "cd20114e7214a942d6aec30bae22bb14"
server_key_iv: "6520c9b70ecca6eb3453b045f1bd4e6e"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.324321 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "c450a736c36d4d82b267866bfa8cfce4"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "cd20114e7214a942d6aec30bae22bb14"
server_key_iv: "6520c9b70ecca6eb3453b045f1bd4e6e"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.328027 4316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:52:20.330448 4337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.331566 4316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 01:52:20.331694 4316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "c450a736c36d4d82b267866bfa8cfce4"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "cd20114e7214a942d6aec30bae22bb14"
server_key_iv: "6520c9b70ecca6eb3453b045f1bd4e6e"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.331804 4316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:20.372634 4316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:20.375895 4316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:20.376118 4316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:20.376792 4316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:20.377756 4316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:20.377823 4316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.377895 4316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:20.377940 4316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.387069 4316 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:40897
I20260809 01:52:20.387095 4512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:40897 every 8 connection(s)
I20260809 01:52:20.388016 4316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:20.397505 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4316
I20260809 01:52:20.397766 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:20.398028 3952 external_mini_cluster.cc:1428] Setting key e70a3b64583e8368fc84e9218408913e
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:20.402926 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:20.390035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:38055 (local address 127.3.220.62:33515)
0809 01:52:20.390185 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:20.390190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:20.391158 (+ 968us) server_negotiation.cc:407] Connection header received
0809 01:52:20.392659 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:20.392662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:20.392710 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:20.392781 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:20.394193 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.394736 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.395982 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.396207 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.396292 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:20.398992 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:20.398996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:20.398998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:20.399021 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:20.400502 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.401124 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.401127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.401128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.401200 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.401631 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.401637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.401641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.401922 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:20.402031 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:20.402545 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 01:52:20.402689 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:52:20.403637 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:20.390474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:33515 (local address 127.3.220.2:38055)
0809 01:52:20.390988 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:20.391046 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:20.392439 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:20.393038 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:20.393047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:20.393483 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:20.394037 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:20.394051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.394842 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.394844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:20.395868 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:20.395879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.396345 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.396348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:20.396490 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:20.397048 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:20.397064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:20.398837 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:20.400620 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:20.400630 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:20.400650 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:20.401026 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:20.401289 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:20.401295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:20.401300 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:20.401492 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:20.402044 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:20.402049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:20.402298 (+ 249us) client_negotiation.cc:769] Sending connection context
0809 01:52:20.402451 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 01:52:20.402708 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":111,"threads_started":1}
I20260809 01:52:20.405062 4513 heartbeater.cc:347] Connected to a master server at 127.3.220.62:33515
I20260809 01:52:20.405411 4513 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:20.406167 4513 heartbeater.cc:511] Master 127.3.220.62:33515 requested a full tablet report, sending...
I20260809 01:52:20.407496 4026 ts_manager.cc:194] Registered new tserver with Master: c450a736c36d4d82b267866bfa8cfce4 (127.3.220.2:40897)
I20260809 01:52:20.408077 4026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:38055
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:20.429033 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:33515
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:20.542018 4522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:20.542275 4522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:20.542320 4522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:20.546358 4522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:20.546411 4522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:20.546499 4522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:20.551553 4522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:33515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:20.552595 4522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:20.553681 4522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:20.563226 4534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:20.563280 4530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:20.563280 4536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:20.563609 4522 server_base.cc:1034] running on GCE node
I20260809 01:52:20.564013 4522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:20.564591 4522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:20.565749 4522 hybrid_clock.cc:648] HybridClock initialized: now 1786240340565721 us; error 43 us; skew 500 ppm
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:20.568364 4522 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:20.569221 4522 webserver.cc:460] Webserver started at http://127.3.220.3:43669/ using document root <none> and password file <none>
I20260809 01:52:20.569841 4522 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:20.569927 4522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:20.570145 4522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:20.572072 4522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "ee3a78903a494874b6b660899b213232"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "9f4f77b767b6a0158b2522225f49e4f6"
server_key_iv: "6208a15368905c53b13eb29f654dce2b"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.572619 4522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "ee3a78903a494874b6b660899b213232"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "9f4f77b767b6a0158b2522225f49e4f6"
server_key_iv: "6208a15368905c53b13eb29f654dce2b"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.576215 4522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 01:52:20.578598 4551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.579790 4522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:20.579914 4522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "ee3a78903a494874b6b660899b213232"
format_stamp: "Formatted at 2026-08-09 01:52:20 on dist-test-slave-gkw7"
server_key: "9f4f77b767b6a0158b2522225f49e4f6"
server_key_iv: "6208a15368905c53b13eb29f654dce2b"
server_key_version: "encryptionkey@0"
I20260809 01:52:20.580015 4522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:20.599838 4522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:20.603060 4522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:20.603295 4522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:20.603875 4522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:20.604696 4522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:20.604764 4522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.604847 4522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:20.604885 4522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:20.614091 4522 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:40419
I20260809 01:52:20.614120 4726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:40419 every 8 connection(s)
I20260809 01:52:20.615021 4522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:20.615923 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4522
I20260809 01:52:20.616048 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:20.616305 3952 external_mini_cluster.cc:1428] Setting key b5655d9d4d9c8a3fa10f08087563cedc
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:20.628685 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:20.616985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:60077 (local address 127.3.220.62:33515)
0809 01:52:20.617127 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:20.617131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:20.618121 (+ 990us) server_negotiation.cc:407] Connection header received
0809 01:52:20.619629 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:20.619632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:20.619685 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:20.619761 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:20.620862 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.621391 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.622544 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.622779 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.622886 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:20.625165 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:20.625169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:20.625172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:20.625239 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:20.626663 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.627188 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.627192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.627194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.627277 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.627617 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.627620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.627622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.627810 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:20.627919 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:20.628394 (+ 475us) server_negotiation.cc:299] Negotiation successful
0809 01:52:20.628511 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:52:20.629521 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:20.617444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:33515 (local address 127.3.220.3:60077)
0809 01:52:20.617982 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:20.618041 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:20.619425 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:20.619931 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:20.619938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:20.620271 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:20.620729 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:20.620741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.621512 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.621515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:20.622421 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:20.622430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.622921 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.622925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:20.623092 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:20.623588 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:20.623603 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:20.624989 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:20.626796 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:20.626801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:20.626816 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:20.627077 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:20.627371 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:20.627375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:20.627377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:20.627518 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:20.627917 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:20.627923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:20.628161 (+ 238us) client_negotiation.cc:769] Sending connection context
0809 01:52:20.628349 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 01:52:20.628669 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":113,"threads_started":1}
I20260809 01:52:20.630659 4727 heartbeater.cc:347] Connected to a master server at 127.3.220.62:33515
I20260809 01:52:20.630914 4727 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:20.631402 4727 heartbeater.cc:511] Master 127.3.220.62:33515 requested a full tablet report, sending...
I20260809 01:52:20.632387 4026 ts_manager.cc:194] Registered new tserver with Master: ee3a78903a494874b6b660899b213232 (127.3.220.3:40419)
I20260809 01:52:20.632939 4026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:60077
I20260809 01:52:20.643092 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:20.672619 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:20.661739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.3.220.1:44991)
0809 01:52:20.662216 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:20.662230 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:20.662255 (+ 25us) server_negotiation.cc:407] Connection header received
0809 01:52:20.662308 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:20.662313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:20.662497 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:20.662616 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:20.663231 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.664293 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.665691 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.666343 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.666476 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:20.669122 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:20.669133 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:20.669146 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:20.669171 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:20.670789 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.671209 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.671214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.671215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.671302 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.671590 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.671594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.671595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.671869 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:20.672034 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:20.672343 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 01:52:20.672481 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":173,"threads_started":1}
W20260809 01:52:20.673620 4182 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33082
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:20.701350 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:20.692436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.3.220.1:44991)
0809 01:52:20.692595 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:20.692600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:20.692764 (+ 164us) server_negotiation.cc:407] Connection header received
0809 01:52:20.692944 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:20.692947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:20.692991 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:20.693075 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:20.693906 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.694427 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.695281 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:20.695513 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:20.695619 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:20.698185 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:20.698189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:20.698191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:20.698216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:20.699706 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.700137 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.700141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.700143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.700234 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:20.700503 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:20.700508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:20.700509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:20.700678 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:20.700837 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:20.701059 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 01:52:20.701175 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 01:52:20.702483 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4117
I20260809 01:52:20.716650 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4316
I20260809 01:52:20.727025 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4522
I20260809 01:52:20.735532 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 3984
2026-08-09T01:52:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2528 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[4748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[4748](info): set up 2 sockets
Aug 09 01:52:20 dist-test-slave-gkw7 krb5kdc[4748](info): commencing operation
krb5kdc: starting...
W20260809 01:52:21.971205 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:22 dist-test-slave-gkw7 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:22.105039 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41029
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:41029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:22.218463 4766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:22.218773 4766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:22.218863 4766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:22.222909 4766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:22.223002 4766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:22.223053 4766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:22.223110 4766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:22.223202 4766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:22.228518 4766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:41029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:22.229700 4766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:22.230782 4766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:22.239542 4774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:22.239538 4772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:22.239663 4776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:22.239810 4766 server_base.cc:1034] running on GCE node
I20260809 01:52:22.240483 4766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:22.241677 4766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:22.242878 4766 hybrid_clock.cc:648] HybridClock initialized: now 1786240342242863 us; error 26 us; skew 500 ppm
Aug 09 01:52:22 dist-test-slave-gkw7 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:22.245759 4766 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:22.246635 4766 webserver.cc:460] Webserver started at http://127.3.220.62:42101/ using document root <none> and password file <none>
I20260809 01:52:22.247243 4766 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:22.247342 4766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:22.247577 4766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:22.249679 4766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "18c560352e064d5e99c3084417d174d8"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "3aca6aa4ce23847e2098565a6a794f96"
server_key_iv: "7e4651798794ead33e1a3641954d67d2"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.250264 4766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "18c560352e064d5e99c3084417d174d8"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "3aca6aa4ce23847e2098565a6a794f96"
server_key_iv: "7e4651798794ead33e1a3641954d67d2"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.254073 4766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:52:22.256525 4787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:22.257894 4766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:22.258041 4766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "18c560352e064d5e99c3084417d174d8"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "3aca6aa4ce23847e2098565a6a794f96"
server_key_iv: "7e4651798794ead33e1a3641954d67d2"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.258154 4766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:22.268715 4766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:22.271394 4766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:22.271633 4766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:22.278553 4766 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:41029
I20260809 01:52:22.278610 4861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:41029 every 8 connection(s)
I20260809 01:52:22.279534 4766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:22.281905 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4766
I20260809 01:52:22.281998 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:22.282274 3952 external_mini_cluster.cc:1428] Setting key 10e0408ee409ae540ab27c70405365bc
I20260809 01:52:22.282501 4862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:22.289824 4862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8: Bootstrap starting.
Aug 09 01:52:22 dist-test-slave-gkw7 krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:22.292268 4862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:22.293186 4862 log.cc:824] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:22.295570 4862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8: No bootstrap required, opened a new log
I20260809 01:52:22.297739 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:22.283742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.3.220.62:41029)
0809 01:52:22.284175 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:22.284194 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:22.284238 (+ 44us) server_negotiation.cc:407] Connection header received
0809 01:52:22.285285 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:22.285310 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:22.285767 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:22.286144 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:22.286836 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.288339 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.289189 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.289907 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.290067 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:22.292561 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:22.292570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:22.292587 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:22.292617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:22.294554 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:22.295006 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:22.295012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:22.295018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:22.295156 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:22.295430 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:22.295433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:22.295434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:22.295952 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:22.296126 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:22.296432 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 01:52:22.296722 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
I20260809 01:52:22.300160 4862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c560352e064d5e99c3084417d174d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41029 } }
I20260809 01:52:22.300421 4862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:22.300472 4862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c560352e064d5e99c3084417d174d8, State: Initialized, Role: FOLLOWER
I20260809 01:52:22.300872 4862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c560352e064d5e99c3084417d174d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41029 } }
I20260809 01:52:22.300993 4862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:22.301039 4862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:22.301110 4862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:22.302048 4862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c560352e064d5e99c3084417d174d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41029 } }
I20260809 01:52:22.302350 4862 leader_election.cc:302] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18c560352e064d5e99c3084417d174d8; no voters:
I20260809 01:52:22.302651 4862 leader_election.cc:288] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:22.302814 4870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:22.303056 4870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [term 1 LEADER]: Becoming Leader. State: Replica: 18c560352e064d5e99c3084417d174d8, State: Running, Role: LEADER
I20260809 01:52:22.303375 4870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c560352e064d5e99c3084417d174d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41029 } }
I20260809 01:52:22.304039 4862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:22.305277 4871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18c560352e064d5e99c3084417d174d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c560352e064d5e99c3084417d174d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41029 } } }
I20260809 01:52:22.305414 4871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:22.305727 4872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18c560352e064d5e99c3084417d174d8. Latest consensus state: current_term: 1 leader_uuid: "18c560352e064d5e99c3084417d174d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c560352e064d5e99c3084417d174d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41029 } } }
I20260809 01:52:22.305830 4872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8 [sys.catalog]: This master's current role is: LEADER
W20260809 01:52:22.308722 4891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:52:22.308815 4891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:52:22.308899 4884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:22.309691 4884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:22.315478 4884 catalog_manager.cc:1347] Generated new cluster ID: 79641e4a1ade4b05b70f817849aaf3ea
I20260809 01:52:22.315544 4884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:22.334503 4884 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:22.335522 4884 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:22.354544 4884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18c560352e064d5e99c3084417d174d8: Generated new TSK 0
I20260809 01:52:22.355356 4884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:22.408464 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41029
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:22.528882 4899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:22.529186 4899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:22.529235 4899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:22.533500 4899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:22.533579 4899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:22.533638 4899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:22.538816 4899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:22.539850 4899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:22.540879 4899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:52:22.550962 4899 server_base.cc:1034] running on GCE node
W20260809 01:52:22.550981 4911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:22.550900 4908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:22.550890 4907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:22.551573 4899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:22.552189 4899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:22.553375 4899 hybrid_clock.cc:648] HybridClock initialized: now 1786240342553338 us; error 50 us; skew 500 ppm
Aug 09 01:52:22 dist-test-slave-gkw7 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:22.556134 4899 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:22.557053 4899 webserver.cc:460] Webserver started at http://127.3.220.1:40343/ using document root <none> and password file <none>
I20260809 01:52:22.557695 4899 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:22.557791 4899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:22.558029 4899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:22.560042 4899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "9837355c3ff743089673bf264feaac40"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "18f12872e790d5425ad14b6871eae949"
server_key_iv: "4db4475125f7d9bb02f53e81e4b68cae"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.560613 4899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "9837355c3ff743089673bf264feaac40"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "18f12872e790d5425ad14b6871eae949"
server_key_iv: "4db4475125f7d9bb02f53e81e4b68cae"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.564666 4899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 01:52:22.567337 4920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:22.568542 4899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:22.568681 4899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "9837355c3ff743089673bf264feaac40"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "18f12872e790d5425ad14b6871eae949"
server_key_iv: "4db4475125f7d9bb02f53e81e4b68cae"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.568801 4899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:22.596791 4899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:22.600205 4899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:22.600440 4899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:22.601114 4899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:22.602027 4899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:22.602128 4899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:22.602224 4899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:22.602277 4899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:22.612463 4899 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:33127
I20260809 01:52:22.612520 5079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:33127 every 8 connection(s)
I20260809 01:52:22.613416 4899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:22.617599 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4899
I20260809 01:52:22.617722 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:22.618026 3952 external_mini_cluster.cc:1428] Setting key 32db0258cdbaff6870fb61425bc0c363
Aug 09 01:52:22 dist-test-slave-gkw7 krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:52:22.628608 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:22.615390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:42833 (local address 127.3.220.62:41029)
0809 01:52:22.615546 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:22.615550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:22.616381 (+ 831us) server_negotiation.cc:407] Connection header received
0809 01:52:22.618001 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:22.618003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:22.618050 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:22.618164 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:22.619571 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.620346 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.621582 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.621824 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.621954 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:22.624785 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:22.624788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:22.624790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:22.624812 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:22.626485 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:22.627027 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:22.627032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:22.627033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:22.627116 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:22.627487 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:22.627490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:22.627492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:22.627682 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:22.627807 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:22.628301 (+ 494us) server_negotiation.cc:299] Negotiation successful
0809 01:52:22.628431 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 01:52:22.629536 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:22.615760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41029 (local address 127.3.220.1:42833)
0809 01:52:22.616224 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:22.616269 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:22.617761 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:22.618362 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:22.618384 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:22.618769 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:22.619425 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:22.619441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.620504 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.620507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:22.621422 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:22.621431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.621995 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.621999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:22.622171 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:22.622833 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:22.622849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:22.624593 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:22.626635 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:22.626640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:22.626656 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:22.626921 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:22.627240 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:22.627243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:22.627245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:22.627389 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:22.627806 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:22.627813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:22.628065 (+ 252us) client_negotiation.cc:769] Sending connection context
0809 01:52:22.628259 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 01:52:22.628582 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20260809 01:52:22.630996 5080 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41029
I20260809 01:52:22.631254 5080 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:52:22.631858 5080 heartbeater.cc:511] Master 127.3.220.62:41029 requested a full tablet report, sending...
I20260809 01:52:22.633730 4806 ts_manager.cc:194] Registered new tserver with Master: 9837355c3ff743089673bf264feaac40 (127.3.220.1:33127)
I20260809 01:52:22.635130 4806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:42833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:22.649685 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41029
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:22.761029 5086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:22.761255 5086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:22.761335 5086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:22.765339 5086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:22.765388 5086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:22.765491 5086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:22.770599 5086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:22.771556 5086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:22.772506 5086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:22.782019 5093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:22.782011 5099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:22.782061 5086 server_base.cc:1034] running on GCE node
W20260809 01:52:22.782011 5092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:22.782614 5086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:22.783246 5086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:22.784431 5086 hybrid_clock.cc:648] HybridClock initialized: now 1786240342784414 us; error 36 us; skew 500 ppm
Aug 09 01:52:22 dist-test-slave-gkw7 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:22.787113 5086 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:22.788038 5086 webserver.cc:460] Webserver started at http://127.3.220.2:35129/ using document root <none> and password file <none>
I20260809 01:52:22.788779 5086 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:22.788872 5086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:22.789088 5086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:22.791028 5086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "0741fa9267e24664ae26a18120bc3145"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "4949fa43eeb5200a0d7ebfc96e6bf9cb"
server_key_iv: "3d1de395a5da92e2bd20a86581cd1fe4"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.791538 5086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "0741fa9267e24664ae26a18120bc3145"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "4949fa43eeb5200a0d7ebfc96e6bf9cb"
server_key_iv: "3d1de395a5da92e2bd20a86581cd1fe4"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.795006 5086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:52:22.797256 5109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:22.798468 5086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:52:22.798560 5086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "0741fa9267e24664ae26a18120bc3145"
format_stamp: "Formatted at 2026-08-09 01:52:22 on dist-test-slave-gkw7"
server_key: "4949fa43eeb5200a0d7ebfc96e6bf9cb"
server_key_iv: "3d1de395a5da92e2bd20a86581cd1fe4"
server_key_version: "encryptionkey@0"
I20260809 01:52:22.798638 5086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:22.822501 5086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:22.825127 5086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:22.825274 5086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:22.825875 5086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:22.826821 5086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:22.826867 5086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:22.826933 5086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:22.826972 5086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:22.835776 5086 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:44371
I20260809 01:52:22.835798 5266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:44371 every 8 connection(s)
I20260809 01:52:22.836643 5086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
Aug 09 01:52:22 dist-test-slave-gkw7 krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240342, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:22.846323 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 5086
I20260809 01:52:22.846433 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:22.846725 3952 external_mini_cluster.cc:1428] Setting key 6363d069c49f0a20275495e34441d3e1
I20260809 01:52:22.850920 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:22.838512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:40597 (local address 127.3.220.62:41029)
0809 01:52:22.838650 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:22.838654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:22.839425 (+ 771us) server_negotiation.cc:407] Connection header received
0809 01:52:22.840796 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:22.840799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:22.840864 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:22.840965 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:22.842218 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.842758 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.843882 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.844112 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.844230 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:22.846836 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:22.846841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:22.846844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:22.846875 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:22.848477 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:22.849084 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:22.849087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:22.849088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:22.849161 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:22.849607 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:22.849612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:22.849618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:22.849787 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:22.849872 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:22.850549 (+ 677us) server_negotiation.cc:299] Negotiation successful
0809 01:52:22.850706 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:52:22.851629 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:22.838832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41029 (local address 127.3.220.2:40597)
0809 01:52:22.839269 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:22.839313 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:22.840572 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:22.841149 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:22.841156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:22.841515 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:22.842057 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:22.842070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.842880 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.842884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:22.843754 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:22.843763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:22.844273 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:22.844277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:22.844441 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:22.844966 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:22.844982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:22.846648 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:22.848592 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:22.848598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:22.848612 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:22.848983 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:22.849280 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:22.849287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:22.849291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:22.849482 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:22.849900 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:22.849909 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:22.850223 (+ 314us) client_negotiation.cc:769] Sending connection context
0809 01:52:22.850451 (+ 228us) client_negotiation.cc:240] Negotiation successful
0809 01:52:22.850759 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
I20260809 01:52:22.852752 5267 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41029
I20260809 01:52:22.853021 5267 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:22.853485 5267 heartbeater.cc:511] Master 127.3.220.62:41029 requested a full tablet report, sending...
I20260809 01:52:22.854571 4806 ts_manager.cc:194] Registered new tserver with Master: 0741fa9267e24664ae26a18120bc3145 (127.3.220.2:44371)
I20260809 01:52:22.855165 4806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:40597
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:22.874830 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41029
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:22.985859 5275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:22.986104 5275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:22.986147 5275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:22.990192 5275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:22.990239 5275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:22.990296 5275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:22.995357 5275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:22.996279 5275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:22.997243 5275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:23.007095 5285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:23.007112 5288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:23.007282 5275 server_base.cc:1034] running on GCE node
W20260809 01:52:23.007089 5292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:23.007699 5275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:23.008375 5275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:23.009573 5275 hybrid_clock.cc:648] HybridClock initialized: now 1786240343009523 us; error 24 us; skew 500 ppm
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:23.012399 5275 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:23.013283 5275 webserver.cc:460] Webserver started at http://127.3.220.3:37321/ using document root <none> and password file <none>
I20260809 01:52:23.013897 5275 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:23.013960 5275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:23.014200 5275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:23.016186 5275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "22fb5596f2fa4e0d88c1edf1fd4514af"
format_stamp: "Formatted at 2026-08-09 01:52:23 on dist-test-slave-gkw7"
server_key: "ef62e9a591961daaf7a995f25a78cb41"
server_key_iv: "b5d26788fe2207e774dcc0c2760dbb32"
server_key_version: "encryptionkey@0"
I20260809 01:52:23.016753 5275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "22fb5596f2fa4e0d88c1edf1fd4514af"
format_stamp: "Formatted at 2026-08-09 01:52:23 on dist-test-slave-gkw7"
server_key: "ef62e9a591961daaf7a995f25a78cb41"
server_key_iv: "b5d26788fe2207e774dcc0c2760dbb32"
server_key_version: "encryptionkey@0"
I20260809 01:52:23.020617 5275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:52:23.023115 5302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:23.024706 5275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 01:52:23.024828 5275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "22fb5596f2fa4e0d88c1edf1fd4514af"
format_stamp: "Formatted at 2026-08-09 01:52:23 on dist-test-slave-gkw7"
server_key: "ef62e9a591961daaf7a995f25a78cb41"
server_key_iv: "b5d26788fe2207e774dcc0c2760dbb32"
server_key_version: "encryptionkey@0"
I20260809 01:52:23.024931 5275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:23.073307 5275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:23.076259 5275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:23.076414 5275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:23.076953 5275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:23.077803 5275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:23.077852 5275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:23.077893 5275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:23.077911 5275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:23.087054 5275 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:39139
I20260809 01:52:23.087083 5471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:39139 every 8 connection(s)
I20260809 01:52:23.088011 5275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:23.091400 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 5275
I20260809 01:52:23.091523 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:23.091773 3952 external_mini_cluster.cc:1428] Setting key c548c38fbbbc3780dd83bfd87052e16b
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:23.101641 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.089770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:50905 (local address 127.3.220.62:41029)
0809 01:52:23.089911 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.089915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.090724 (+ 809us) server_negotiation.cc:407] Connection header received
0809 01:52:23.092272 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.092277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.092335 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.092430 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:23.094061 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.094617 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.095694 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.095906 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.095981 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.098226 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:23.098240 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:23.098242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:23.098266 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:23.099546 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.100015 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.100017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.100019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.100092 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.100395 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.100397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.100398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.100557 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:23.100627 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.101370 (+ 743us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.101469 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:52:23.102594 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.090118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41029 (local address 127.3.220.3:50905)
0809 01:52:23.090563 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:23.090605 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:23.092002 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:23.092679 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:23.092690 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:23.093189 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:23.093900 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.093921 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.094729 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.094731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.095591 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.095600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.096027 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.096029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.096163 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.096681 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:23.096697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:23.098080 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:23.099650 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:23.099654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:23.099674 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:23.099922 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:23.100168 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:23.100171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:23.100173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:23.100308 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:23.100799 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:23.100804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:23.101031 (+ 227us) client_negotiation.cc:769] Sending connection context
0809 01:52:23.101239 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 01:52:23.101691 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260809 01:52:23.103773 5472 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41029
I20260809 01:52:23.103978 5472 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:23.104493 5472 heartbeater.cc:511] Master 127.3.220.62:41029 requested a full tablet report, sending...
I20260809 01:52:23.105700 4805 ts_manager.cc:194] Registered new tserver with Master: 22fb5596f2fa4e0d88c1edf1fd4514af (127.3.220.3:39139)
I20260809 01:52:23.106302 4805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:50905
I20260809 01:52:23.117656 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:52:23.129139 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.121829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.3.220.62:41029)
0809 01:52:23.122066 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.122070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.122171 (+ 101us) server_negotiation.cc:407] Connection header received
0809 01:52:23.122273 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.122275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.122311 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.122379 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:23.123067 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.123560 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.124364 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.124568 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.124640 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.126160 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:23.126163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:23.126165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:23.126213 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:23.127663 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.128052 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.128055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.128056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.128127 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.128366 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.128368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.128369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.128552 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:23.128704 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.128874 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.129014 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260809 01:52:23.134711 4805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:52:23.137925 4805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:23.153486 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.147367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:39139 (local address 127.0.0.1:34640)
0809 01:52:23.147875 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:23.147923 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:23.148054 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:23.149143 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:23.149149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:23.149174 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:23.149306 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.149313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.150867 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.150871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.152037 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.152047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.152856 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.152859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.152987 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.153029 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 01:52:23.153149 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 01:52:23.153267 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260809 01:52:23.153518 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.147981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:44371 (local address 127.0.0.1:50538)
0809 01:52:23.148314 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:23.148333 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:23.148424 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:23.149286 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:23.149288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:23.149304 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:23.149412 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.149418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.150913 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.150916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.152005 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.152014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.153059 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.153063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.153146 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.153182 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 01:52:23.153239 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 01:52:23.153292 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":33,"thread_start_us":78,"threads_started":1}
I20260809 01:52:23.153569 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.147261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:33127 (local address 127.0.0.1:60038)
0809 01:52:23.147890 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:23.148428 (+ 538us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:23.148519 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:23.149461 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:23.149463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:23.149479 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:23.149589 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.149598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.150560 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.150566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.151968 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.151979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.153156 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.153159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.153251 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.153271 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 01:52:23.153345 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 01:52:23.153411 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":47,"thread_start_us":70,"threads_started":1}
I20260809 01:52:23.153679 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.147392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.3.220.1:33127)
0809 01:52:23.148870 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.148877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.148898 (+ 21us) server_negotiation.cc:407] Connection header received
0809 01:52:23.148976 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.148983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.149300 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.149453 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:23.149713 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.150443 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.152101 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.152917 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.153036 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.153172 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.153396 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.153494 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":126,"threads_started":1}
I20260809 01:52:23.154035 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.148033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.3.220.3:39139)
0809 01:52:23.148673 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.148685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.148703 (+ 18us) server_negotiation.cc:407] Connection header received
0809 01:52:23.148766 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.148770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.148924 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.149089 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:23.149449 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.150725 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.152156 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.152720 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.152859 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.153073 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.153710 (+ 637us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.153854 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":129,"threads_started":1}
I20260809 01:52:23.154630 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.148142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.3.220.2:44371)
0809 01:52:23.148584 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.148598 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.148622 (+ 24us) server_negotiation.cc:407] Connection header received
0809 01:52:23.148720 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.148727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.148924 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.149089 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:23.149582 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.150725 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.152174 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.152925 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.154225 (+ 1300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.154358 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.154403 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.154486 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":117,"threads_started":1}
I20260809 01:52:23.156877 4983 tablet_service.cc:1467] Processing CreateTablet for tablet c83610f0378043e6bea93989623ca578 (DEFAULT_TABLE table=test-table [id=b871e42ac0444e3d9afaab5e537b1c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:23.156945 5366 tablet_service.cc:1467] Processing CreateTablet for tablet c83610f0378043e6bea93989623ca578 (DEFAULT_TABLE table=test-table [id=b871e42ac0444e3d9afaab5e537b1c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:23.157241 5172 tablet_service.cc:1467] Processing CreateTablet for tablet c83610f0378043e6bea93989623ca578 (DEFAULT_TABLE table=test-table [id=b871e42ac0444e3d9afaab5e537b1c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:23.158144 4983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83610f0378043e6bea93989623ca578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:23.158175 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83610f0378043e6bea93989623ca578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:23.158191 5172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c83610f0378043e6bea93989623ca578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:23.164839 5496 tablet_bootstrap.cc:492] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40: Bootstrap starting.
I20260809 01:52:23.165722 5498 tablet_bootstrap.cc:492] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af: Bootstrap starting.
I20260809 01:52:23.165722 5501 tablet_bootstrap.cc:492] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145: Bootstrap starting.
I20260809 01:52:23.167546 5496 tablet_bootstrap.cc:654] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:23.167897 5498 tablet_bootstrap.cc:654] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:23.168340 5496 log.cc:824] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:23.168615 5501 tablet_bootstrap.cc:654] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:23.169348 5498 log.cc:824] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:23.169348 5501 log.cc:824] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:23.171029 5501 tablet_bootstrap.cc:492] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145: No bootstrap required, opened a new log
I20260809 01:52:23.171037 5496 tablet_bootstrap.cc:492] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40: No bootstrap required, opened a new log
I20260809 01:52:23.171227 5501 ts_tablet_manager.cc:1397] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:52:23.171236 5496 ts_tablet_manager.cc:1397] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 01:52:23.172255 5498 tablet_bootstrap.cc:492] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af: No bootstrap required, opened a new log
I20260809 01:52:23.172422 5498 ts_tablet_manager.cc:1397] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 01:52:23.174736 5501 raft_consensus.cc:348] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.175056 5501 raft_consensus.cc:374] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:23.175112 5501 raft_consensus.cc:729] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0741fa9267e24664ae26a18120bc3145, State: Initialized, Role: FOLLOWER
I20260809 01:52:23.175748 5501 consensus_queue.cc:260] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.175729 5496 raft_consensus.cc:348] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.175969 5496 raft_consensus.cc:374] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:23.176017 5496 raft_consensus.cc:729] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9837355c3ff743089673bf264feaac40, State: Initialized, Role: FOLLOWER
I20260809 01:52:23.176169 5498 raft_consensus.cc:348] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.176407 5498 raft_consensus.cc:374] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:23.176465 5498 raft_consensus.cc:729] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fb5596f2fa4e0d88c1edf1fd4514af, State: Initialized, Role: FOLLOWER
I20260809 01:52:23.176698 5267 heartbeater.cc:503] Master 127.3.220.62:41029 was elected leader, sending a full tablet report...
I20260809 01:52:23.176833 5501 ts_tablet_manager.cc:1428] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:52:23.176867 5498 consensus_queue.cc:260] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.177492 5472 heartbeater.cc:503] Master 127.3.220.62:41029 was elected leader, sending a full tablet report...
I20260809 01:52:23.176471 5496 consensus_queue.cc:260] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.177856 5498 ts_tablet_manager.cc:1428] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260809 01:52:23.178417 5080 heartbeater.cc:503] Master 127.3.220.62:41029 was elected leader, sending a full tablet report...
I20260809 01:52:23.179500 5496 ts_tablet_manager.cc:1428] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260809 01:52:23.231405 5508 raft_consensus.cc:482] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:23.231664 5508 raft_consensus.cc:504] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.232846 5508 leader_election.cc:288] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9837355c3ff743089673bf264feaac40 (127.3.220.1:33127), 0741fa9267e24664ae26a18120bc3145 (127.3.220.2:44371)
I20260809 01:52:23.238122 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.233702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:38133 (local address 127.3.220.2:44371)
0809 01:52:23.233875 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.233880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.234048 (+ 168us) server_negotiation.cc:407] Connection header received
0809 01:52:23.234192 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.234195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.234250 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.234337 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:23.234597 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.235253 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.236797 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.237388 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.237481 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.237559 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.237914 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.237989 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 01:52:23.238122 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.233320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:47935 (local address 127.3.220.1:33127)
0809 01:52:23.233454 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.233475 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.233494 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:52:23.233576 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.233580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.233698 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.233834 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:23.234219 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.235006 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.236666 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.237344 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.237466 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.237559 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.237914 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.237998 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 01:52:23.238160 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.233185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:33127 (local address 127.3.220.3:47935)
0809 01:52:23.233333 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:23.233352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:23.233463 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:23.233902 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:23.233906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:23.233925 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:23.234077 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.234087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.235129 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.235133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.236532 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.236543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.237505 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.237509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.237710 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.237795 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 01:52:23.237880 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 01:52:23.237996 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":127}
I20260809 01:52:23.238169 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.233563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:44371 (local address 127.3.220.3:38133)
0809 01:52:23.233945 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:23.233959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:23.234061 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:23.234361 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:23.234363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:23.234379 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:23.234482 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.234487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.235363 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.235366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.236668 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:23.236679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.237505 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.237509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:23.237686 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.237786 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 01:52:23.237880 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 01:52:23.237987 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":44,"thread_start_us":154,"threads_started":1}
I20260809 01:52:23.238932 5012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83610f0378043e6bea93989623ca578" candidate_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9837355c3ff743089673bf264feaac40" is_pre_election: true
I20260809 01:52:23.238940 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83610f0378043e6bea93989623ca578" candidate_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0741fa9267e24664ae26a18120bc3145" is_pre_election: true
I20260809 01:52:23.239311 5200 raft_consensus.cc:2454] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fb5596f2fa4e0d88c1edf1fd4514af in term 0.
I20260809 01:52:23.239850 5303 leader_election.cc:302] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0741fa9267e24664ae26a18120bc3145, 22fb5596f2fa4e0d88c1edf1fd4514af; no voters:
I20260809 01:52:23.240154 5508 raft_consensus.cc:2781] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:23.240238 5508 raft_consensus.cc:482] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:23.240271 5508 raft_consensus.cc:3037] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:23.241358 5012 raft_consensus.cc:2454] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22fb5596f2fa4e0d88c1edf1fd4514af in term 0.
I20260809 01:52:23.241336 5508 raft_consensus.cc:504] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.241786 5508 leader_election.cc:288] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [CANDIDATE]: Term 1 election: Requested vote from peers 9837355c3ff743089673bf264feaac40 (127.3.220.1:33127), 0741fa9267e24664ae26a18120bc3145 (127.3.220.2:44371)
I20260809 01:52:23.242103 5012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83610f0378043e6bea93989623ca578" candidate_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9837355c3ff743089673bf264feaac40"
I20260809 01:52:23.242247 5012 raft_consensus.cc:3037] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:23.242564 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c83610f0378043e6bea93989623ca578" candidate_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0741fa9267e24664ae26a18120bc3145"
I20260809 01:52:23.242699 5200 raft_consensus.cc:3037] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:23.243631 5012 raft_consensus.cc:2454] T c83610f0378043e6bea93989623ca578 P 9837355c3ff743089673bf264feaac40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fb5596f2fa4e0d88c1edf1fd4514af in term 1.
I20260809 01:52:23.243952 5303 leader_election.cc:302] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fb5596f2fa4e0d88c1edf1fd4514af, 9837355c3ff743089673bf264feaac40; no voters:
I20260809 01:52:23.244060 5200 raft_consensus.cc:2454] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22fb5596f2fa4e0d88c1edf1fd4514af in term 1.
I20260809 01:52:23.244238 5508 raft_consensus.cc:2781] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:23.244493 5508 raft_consensus.cc:686] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [term 1 LEADER]: Becoming Leader. State: Replica: 22fb5596f2fa4e0d88c1edf1fd4514af, State: Running, Role: LEADER
I20260809 01:52:23.244831 5508 consensus_queue.cc:237] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } }
I20260809 01:52:23.248338 4806 catalog_manager.cc:5495] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af reported cstate change: term changed from 0 to 1, leader changed from <none> to 22fb5596f2fa4e0d88c1edf1fd4514af (127.3.220.3). New cstate: current_term: 1 leader_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9837355c3ff743089673bf264feaac40" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 33127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22fb5596f2fa4e0d88c1edf1fd4514af" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0741fa9267e24664ae26a18120bc3145" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44371 } health_report { overall_health: UNKNOWN } } }
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:23.284443 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.267811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.3.220.1:33127)
0809 01:52:23.267981 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.267986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.268196 (+ 210us) server_negotiation.cc:407] Connection header received
0809 01:52:23.268359 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.268365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.268413 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.268505 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:23.269223 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.269835 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.270672 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.270903 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.271008 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.274016 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:23.274031 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:23.274052 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:23.274077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:23.276367 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.276864 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.276871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.276873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.276972 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.277255 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.277258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.277260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.277584 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:23.277789 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.284119 (+ 6330us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.284273 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:23.320645 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.308030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.3.220.1:33127)
0809 01:52:23.308198 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.308203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.308350 (+ 147us) server_negotiation.cc:407] Connection header received
0809 01:52:23.308536 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.308539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.308582 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.308670 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:23.309348 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.309939 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.310796 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.311020 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.311135 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.317482 (+ 6347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:23.317485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:23.317487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:23.317511 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:23.318912 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.319337 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.319341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.319342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.319455 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.319745 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.319749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.319750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.319932 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:23.320066 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.320354 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.320478 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260809 01:52:23.337819 5270 tablet.cc:2369] T c83610f0378043e6bea93989623ca578 P 0741fa9267e24664ae26a18120bc3145: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 01:52:23.338922 5474 tablet.cc:2369] T c83610f0378043e6bea93989623ca578 P 22fb5596f2fa4e0d88c1edf1fd4514af: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:23.347263 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:23.338985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.3.220.1:33127)
0809 01:52:23.339116 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:23.339121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:23.339366 (+ 245us) server_negotiation.cc:407] Connection header received
0809 01:52:23.339571 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:23.339573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:23.339617 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:23.339688 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:23.340343 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.340831 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.341689 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:23.341919 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:23.341997 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:23.344232 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:23.344236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:23.344238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:23.344267 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:23.345760 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.346155 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.346159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.346160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.346249 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:23.346489 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:23.346492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:23.346493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:23.346699 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:23.346814 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:23.347023 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 01:52:23.347134 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 01:52:23.348737 4956 tablet_service.cc:2908] Rejecting scan request for tablet c83610f0378043e6bea93989623ca578: Uninitialized: clean time has not yet been initialized
I20260809 01:52:23.351065 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4899
I20260809 01:52:23.361737 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 5086
I20260809 01:52:23.387158 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 5275
I20260809 01:52:23.397393 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4766
2026-08-09T01:52:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2665 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[5534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[5534](info): set up 2 sockets
Aug 09 01:52:23 dist-test-slave-gkw7 krb5kdc[5534](info): commencing operation
krb5kdc: starting...
W20260809 01:52:24.741216 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:24 dist-test-slave-gkw7 krb5kdc[5534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:24.868259 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:34587
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:34587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:24.980396 5554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:24.980652 5554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:24.980691 5554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:24.984742 5554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:24.984797 5554 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 01:52:24.984813 5554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:24.984831 5554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:24.984880 5554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:24.984912 5554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:24.990219 5554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:34587
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:34587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:24.991238 5554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:24.992281 5554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:25.001566 5566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:25.001668 5554 server_base.cc:1034] running on GCE node
W20260809 01:52:25.001595 5563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:25.001595 5562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:25.002419 5554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:25.003517 5554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:25.004715 5554 hybrid_clock.cc:648] HybridClock initialized: now 1786240345004682 us; error 45 us; skew 500 ppm
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:25.007664 5554 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:25.008574 5554 webserver.cc:460] Webserver started at http://127.3.220.62:39951/ using document root <none> and password file <none>
I20260809 01:52:25.009152 5554 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:25.009245 5554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:25.009485 5554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:25.011617 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "65dafd2a63094de897507a2fd0292e9e"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "c32127f59e5994d5d16c2224e8005d1e"
server_key_iv: "c8c8a948e5845ad1d327cb6f9d2fa779"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.012210 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "65dafd2a63094de897507a2fd0292e9e"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "c32127f59e5994d5d16c2224e8005d1e"
server_key_iv: "c8c8a948e5845ad1d327cb6f9d2fa779"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.016057 5554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 01:52:25.018492 5575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:25.019716 5554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:25.019871 5554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "65dafd2a63094de897507a2fd0292e9e"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "c32127f59e5994d5d16c2224e8005d1e"
server_key_iv: "c8c8a948e5845ad1d327cb6f9d2fa779"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.019980 5554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:25.059513 5554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:25.062880 5554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:25.063094 5554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:25.071707 5554 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:34587
I20260809 01:52:25.071740 5661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:34587 every 8 connection(s)
I20260809 01:52:25.072676 5554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:25.075001 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 5554
I20260809 01:52:25.075094 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:25.075390 3952 external_mini_cluster.cc:1428] Setting key e90b0ddfb473befffb46080ec22a7734
I20260809 01:52:25.075647 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:25.083005 5662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e: Bootstrap starting.
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:25.085646 5662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:25.087044 5662 log.cc:824] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:25.089632 5662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e: No bootstrap required, opened a new log
I20260809 01:52:25.090802 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.076835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.3.220.62:34587)
0809 01:52:25.077254 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.077276 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.077318 (+ 42us) server_negotiation.cc:407] Connection header received
0809 01:52:25.078086 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.078108 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.078526 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.078909 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:25.079566 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.081071 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.082001 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.082659 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.082832 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.085447 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:25.085457 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:25.085476 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:25.085515 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:25.087576 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.088113 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.088119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.088125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.088275 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.088546 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.088549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.088551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.089095 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:25.089302 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.089492 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.089840 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":108,"threads_started":1}
I20260809 01:52:25.093338 5662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dafd2a63094de897507a2fd0292e9e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 34587 } }
I20260809 01:52:25.093606 5662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:25.093659 5662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65dafd2a63094de897507a2fd0292e9e, State: Initialized, Role: FOLLOWER
I20260809 01:52:25.094071 5662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dafd2a63094de897507a2fd0292e9e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 34587 } }
I20260809 01:52:25.094204 5662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:25.094287 5662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:25.094393 5662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:25.095348 5662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dafd2a63094de897507a2fd0292e9e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 34587 } }
I20260809 01:52:25.095745 5662 leader_election.cc:302] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65dafd2a63094de897507a2fd0292e9e; no voters:
I20260809 01:52:25.096154 5662 leader_election.cc:288] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:25.096263 5668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:25.096522 5668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [term 1 LEADER]: Becoming Leader. State: Replica: 65dafd2a63094de897507a2fd0292e9e, State: Running, Role: LEADER
I20260809 01:52:25.096833 5668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dafd2a63094de897507a2fd0292e9e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 34587 } }
I20260809 01:52:25.097419 5662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:25.098619 5669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65dafd2a63094de897507a2fd0292e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dafd2a63094de897507a2fd0292e9e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 34587 } } }
I20260809 01:52:25.098765 5669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:25.098969 5670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65dafd2a63094de897507a2fd0292e9e. Latest consensus state: current_term: 1 leader_uuid: "65dafd2a63094de897507a2fd0292e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dafd2a63094de897507a2fd0292e9e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 34587 } } }
I20260809 01:52:25.099062 5670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e [sys.catalog]: This master's current role is: LEADER
W20260809 01:52:25.102257 5694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:52:25.102355 5694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:52:25.102437 5688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:25.103166 5688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:25.108538 5688 catalog_manager.cc:1347] Generated new cluster ID: 4e63a6096eb54d689e262b72f9ffb769
I20260809 01:52:25.108605 5688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:25.124347 5688 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:25.125329 5688 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:25.135941 5688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e: Generated new TSK 0
I20260809 01:52:25.136631 5688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:25.164812 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:34587
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:25.284051 5702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:25.284461 5702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:25.284567 5702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:25.292448 5702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:25.292536 5702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:25.292641 5702 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:52:25.292721 5702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:25.301254 5702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:34587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:25.302743 5702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:25.304072 5702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:25.313817 5714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:25.313803 5712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:25.313787 5711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:25.314426 5702 server_base.cc:1034] running on GCE node
I20260809 01:52:25.314860 5702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:25.315537 5702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:25.316711 5702 hybrid_clock.cc:648] HybridClock initialized: now 1786240345316653 us; error 40 us; skew 500 ppm
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:25.319696 5702 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:25.320536 5702 webserver.cc:460] Webserver started at http://127.3.220.1:33467/ using document root <none> and password file <none>
I20260809 01:52:25.321103 5702 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:25.321197 5702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:25.321432 5702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:25.323587 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "223b5686a79e49a2ad43a3e6da8d1250"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "f77befbd64816e7c075d23703046a3b4"
server_key_iv: "0ce0fa035096307f1c26b9726f8351c5"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.324215 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "223b5686a79e49a2ad43a3e6da8d1250"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "f77befbd64816e7c075d23703046a3b4"
server_key_iv: "0ce0fa035096307f1c26b9726f8351c5"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.328382 5702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 01:52:25.330922 5722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:25.332311 5702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:25.332455 5702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "223b5686a79e49a2ad43a3e6da8d1250"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "f77befbd64816e7c075d23703046a3b4"
server_key_iv: "0ce0fa035096307f1c26b9726f8351c5"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.332572 5702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:25.347563 5702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:25.350495 5702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:25.350701 5702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:25.352083 5702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:25.352161 5702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:25.352232 5702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:25.352283 5702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:25.363910 5702 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:43689
I20260809 01:52:25.363940 5885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:43689 every 8 connection(s)
I20260809 01:52:25.364845 5702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:25.366958 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.353128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:53927 (local address 127.3.220.62:34587)
0809 01:52:25.353280 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.353284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.354343 (+ 1059us) server_negotiation.cc:407] Connection header received
0809 01:52:25.355500 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.355506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.355580 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.355691 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:25.357316 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.358176 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.359546 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.359804 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.359923 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.363091 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:25.363094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:25.363096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:25.363120 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:25.364675 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.365345 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.365349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.365350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.365444 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.365874 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.365880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.365884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.366091 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:25.366182 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.366720 (+ 538us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.366838 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 01:52:25.367694 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.353697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:34587 (local address 127.3.220.1:53927)
0809 01:52:25.354200 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.354240 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.355202 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.355918 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.355927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.356423 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:25.357118 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.357142 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.358302 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.358305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.359414 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.359426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.359989 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.359993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.360173 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.360747 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:25.360766 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:25.362912 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:25.364792 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.364801 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.364817 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.365228 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.365583 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.365585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.365587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.365763 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.366202 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:25.366208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:25.366462 (+ 254us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.366612 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.366856 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260809 01:52:25.368976 5887 heartbeater.cc:347] Connected to a master server at 127.3.220.62:34587
I20260809 01:52:25.369189 5887 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:25.369762 5887 heartbeater.cc:511] Master 127.3.220.62:34587 requested a full tablet report, sending...
I20260809 01:52:25.371294 5599 ts_manager.cc:194] Registered new tserver with Master: 223b5686a79e49a2ad43a3e6da8d1250 (127.3.220.1:43689)
I20260809 01:52:25.372828 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 5702
I20260809 01:52:25.372956 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:25.372944 5599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:53927
I20260809 01:52:25.373260 3952 external_mini_cluster.cc:1428] Setting key dd51c5974eab44562d77095a1a6c899e
I20260809 01:52:25.379601 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.371508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:60575 (local address 127.3.220.62:34587)
0809 01:52:25.371628 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.371632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.371828 (+ 196us) server_negotiation.cc:407] Connection header received
0809 01:52:25.371949 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.371951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.371988 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.372053 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:25.372584 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.373207 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.374423 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.374646 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.374762 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.376048 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:25.376051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:25.376053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:25.376074 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:25.377690 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.378156 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.378161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.378165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.378263 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.378575 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.378578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.378580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.378833 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:25.378929 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.379301 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.379440 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 01:52:25.379628 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.371451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:34587 (local address 127.3.220.1:60575)
0809 01:52:25.371738 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.371754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.371849 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.372102 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.372105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.372316 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:25.372442 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.372449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.373405 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.373409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.374281 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.374291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.374773 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.374777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.374885 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.375397 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:25.375412 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:25.375907 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:25.377813 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.377816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.377817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.378046 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.378380 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.378383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.378385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.378472 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.378963 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:25.378966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:25.379058 (+ 92us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.379213 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.379376 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:25.402675 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:34587
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:25.515057 5900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:25.515283 5900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:25.515341 5900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:25.519402 5900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:25.519452 5900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:25.519511 5900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:52:25.519593 5900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:25.524983 5900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:34587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:25.526034 5900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:25.527024 5900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:25.536907 5917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:25.536914 5914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:25.537010 5900 server_base.cc:1034] running on GCE node
W20260809 01:52:25.536955 5915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:25.537636 5900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:25.538234 5900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:25.539398 5900 hybrid_clock.cc:648] HybridClock initialized: now 1786240345539383 us; error 25 us; skew 500 ppm
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:25.542160 5900 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:25.542989 5900 webserver.cc:460] Webserver started at http://127.3.220.2:34865/ using document root <none> and password file <none>
I20260809 01:52:25.543540 5900 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:25.543601 5900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:25.543855 5900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:25.545900 5900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "f1d452a92fde4d6dbf85704d9b27cbc8"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "52f7a8e3bb73227caff6af91579b784a"
server_key_iv: "e8a8f2e84651691bad1b524d4cca581e"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.546461 5900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "f1d452a92fde4d6dbf85704d9b27cbc8"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "52f7a8e3bb73227caff6af91579b784a"
server_key_iv: "e8a8f2e84651691bad1b524d4cca581e"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.550058 5900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:25.552400 5925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:25.553648 5900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:25.553776 5900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "f1d452a92fde4d6dbf85704d9b27cbc8"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "52f7a8e3bb73227caff6af91579b784a"
server_key_iv: "e8a8f2e84651691bad1b524d4cca581e"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.553885 5900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:25.568212 5900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:25.571376 5900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:25.571576 5900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:25.573027 5900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:25.573098 5900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:25.573165 5900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:25.573218 5900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:25.584372 5900 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:44929
I20260809 01:52:25.584460 6099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:44929 every 8 connection(s)
I20260809 01:52:25.585552 5900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:25.588536 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.574065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:44373 (local address 127.3.220.62:34587)
0809 01:52:25.574270 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.574275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.575342 (+ 1067us) server_negotiation.cc:407] Connection header received
0809 01:52:25.576543 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.576547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.576644 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.576751 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:25.578360 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.578959 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.580408 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.580844 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.580972 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.584779 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:25.584783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:25.584786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:25.584814 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:25.586510 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.587033 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.587036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.587037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.587115 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.587441 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.587443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.587444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.587612 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:25.587690 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.588212 (+ 522us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.588351 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260809 01:52:25.589501 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.574584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:34587 (local address 127.3.220.2:44373)
0809 01:52:25.575183 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.575227 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.576269 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.576965 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.576975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.577450 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:25.578202 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.578224 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.579198 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.579201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.580273 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.580283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.581017 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.581022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.581264 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.582807 (+ 1543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:25.582825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:25.584548 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:25.586627 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.586636 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.586652 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.586936 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.587205 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.587208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.587209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.587355 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.587706 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:25.587711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:25.587939 (+ 228us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.588143 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.588482 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":153,"threads_started":1}
I20260809 01:52:25.589509 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 5900
I20260809 01:52:25.589664 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:25.589977 3952 external_mini_cluster.cc:1428] Setting key 78dd82c99159085685dc85bb7db15260
I20260809 01:52:25.591148 6100 heartbeater.cc:347] Connected to a master server at 127.3.220.62:34587
I20260809 01:52:25.591342 6100 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:25.592015 6100 heartbeater.cc:511] Master 127.3.220.62:34587 requested a full tablet report, sending...
I20260809 01:52:25.593214 5599 ts_manager.cc:194] Registered new tserver with Master: f1d452a92fde4d6dbf85704d9b27cbc8 (127.3.220.2:44929)
I20260809 01:52:25.594111 5599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:44373
I20260809 01:52:25.601976 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.593904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:47169 (local address 127.3.220.62:34587)
0809 01:52:25.594051 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.594058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.594072 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:25.594111 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.594117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.594168 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.594245 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:25.594903 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.595527 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.596321 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.596548 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.596647 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.598341 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:25.598344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:25.598346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:25.598369 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:25.600162 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.600671 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.600674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.600675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.600759 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.601062 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.601064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.601065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.601314 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:25.601401 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.601752 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.601862 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 01:52:25.602102 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.593174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:34587 (local address 127.3.220.2:47169)
0809 01:52:25.593483 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.593495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.593626 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.594374 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.594378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.594612 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:25.594759 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.594767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.595660 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.595662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.596210 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.596217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.596676 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.596683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.596813 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.597335 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:25.597355 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:25.597901 (+ 546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:25.600286 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.600291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.600293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.600560 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.600855 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.600859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.600861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.600963 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.601424 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:25.601427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:25.601572 (+ 145us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.601706 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.601877 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:25.620416 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:34587
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:25.737072 6110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:25.737380 6110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:25.737457 6110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:25.741406 6110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:25.741509 6110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:25.741601 6110 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:52:25.741693 6110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:25.746843 6110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:34587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:25.747866 6110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:25.748904 6110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:25.758893 6118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:25.758860 6119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:25.758968 6110 server_base.cc:1034] running on GCE node
W20260809 01:52:25.758872 6124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:25.759627 6110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:25.760214 6110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:25.761408 6110 hybrid_clock.cc:648] HybridClock initialized: now 1786240345761393 us; error 26 us; skew 500 ppm
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:25.764111 6110 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:25.764909 6110 webserver.cc:460] Webserver started at http://127.3.220.3:41071/ using document root <none> and password file <none>
I20260809 01:52:25.765412 6110 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:25.765467 6110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:25.765702 6110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:25.767637 6110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "20cdd6c5bfd74a4f9101f4e86c73894d"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "a7a5e92015553363d36f67b400daa1fc"
server_key_iv: "85507c5c060ca063df7417b29c29c76e"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.768144 6110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "20cdd6c5bfd74a4f9101f4e86c73894d"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "a7a5e92015553363d36f67b400daa1fc"
server_key_iv: "85507c5c060ca063df7417b29c29c76e"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.771845 6110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:52:25.774142 6137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:25.775319 6110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:25.775440 6110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "20cdd6c5bfd74a4f9101f4e86c73894d"
format_stamp: "Formatted at 2026-08-09 01:52:25 on dist-test-slave-gkw7"
server_key: "a7a5e92015553363d36f67b400daa1fc"
server_key_iv: "85507c5c060ca063df7417b29c29c76e"
server_key_version: "encryptionkey@0"
I20260809 01:52:25.775543 6110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:25.792692 6110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:25.795921 6110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:25.796068 6110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:25.797578 6110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:25.797647 6110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:25.797720 6110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:25.797766 6110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:25.809253 6110 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:42025
I20260809 01:52:25.809513 6295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:42025 every 8 connection(s)
I20260809 01:52:25.810325 6110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:25.812060 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.798472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:36317 (local address 127.3.220.62:34587)
0809 01:52:25.798601 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.798607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.799633 (+ 1026us) server_negotiation.cc:407] Connection header received
0809 01:52:25.800975 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.800978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.801062 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.801166 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:25.802829 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.803782 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.804910 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.805150 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.805269 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.808329 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:25.808332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:25.808334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:25.808360 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:25.810007 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.810587 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.810589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.810591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.810668 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.811100 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.811104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.811106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.811321 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:25.811401 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.811828 (+ 427us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.811944 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:52:25.812983 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.798936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:34587 (local address 127.3.220.3:36317)
0809 01:52:25.799446 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.799515 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.800681 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.801386 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.801400 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.801933 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:25.802649 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.802672 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.803903 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.803906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.804776 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.804786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.805325 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.805330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.805496 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.806181 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:25.806198 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:25.808133 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:25.810123 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.810128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.810142 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.810491 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.810795 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.810798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.810800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.810967 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.811398 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:25.811403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:25.811616 (+ 213us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.811774 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.812023 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":98,"threads_started":1}
I20260809 01:52:25.814554 6298 heartbeater.cc:347] Connected to a master server at 127.3.220.62:34587
I20260809 01:52:25.814755 6298 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:25.815342 6298 heartbeater.cc:511] Master 127.3.220.62:34587 requested a full tablet report, sending...
I20260809 01:52:25.816991 5599 ts_manager.cc:194] Registered new tserver with Master: 20cdd6c5bfd74a4f9101f4e86c73894d (127.3.220.3:42025)
I20260809 01:52:25.817036 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 6110
I20260809 01:52:25.817159 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:25.817448 3952 external_mini_cluster.cc:1428] Setting key 8d8fc30a3f7f1949f9454d9e2af08bd6
I20260809 01:52:25.818018 5599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:36317
I20260809 01:52:25.819908 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:52:25.826838 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.817737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:55465 (local address 127.3.220.62:34587)
0809 01:52:25.817855 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.817859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.818435 (+ 576us) server_negotiation.cc:407] Connection header received
0809 01:52:25.818712 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.818715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.818769 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.818848 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:25.819395 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.820118 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.821253 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.821455 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.821590 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.823005 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:25.823012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:25.823018 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:25.823051 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:25.824984 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.825433 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.825440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.825444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.825570 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.825867 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.825873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.825877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.826125 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:25.826236 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.826613 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.826713 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 01:52:25.827023 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.817983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:34587 (local address 127.3.220.3:55465)
0809 01:52:25.818334 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.818352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.818477 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.818874 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.818877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.819115 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:25.819256 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.819264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.820252 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.820256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.821115 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.821130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.821613 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.821620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.821748 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.822321 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:25.822342 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:25.822845 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:25.825099 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.825101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.825103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.825308 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.825676 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:25.825679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:25.825680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:25.825760 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:25.826256 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:25.826258 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:25.826419 (+ 161us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.826553 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.826738 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 01:52:25 dist-test-slave-gkw7 krb5kdc[5534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:25.853427 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.844396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.3.220.62:34587)
0809 01:52:25.844575 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.844580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.844593 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:52:25.844640 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.844642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.844685 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.844760 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:25.845891 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.846480 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.847374 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.847625 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.847737 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.850448 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:25.850451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:25.850453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:25.850479 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:25.851848 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.852253 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.852255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.852257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.852329 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:25.852580 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:25.852582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:25.852583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:25.852766 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:25.852863 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.853196 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.853301 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 01:52:25.856516 5599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:52:25.859771 5599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:25.874074 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.868431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:44929 (local address 127.0.0.1:47236)
0809 01:52:25.869227 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.869240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.869312 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.869923 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.869929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.869951 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:25.870158 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.870170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.871426 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.871433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.872691 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.872700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.873511 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.873515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.873643 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.873683 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.873808 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.873930 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":556,"spinlock_wait_cycles":896,"thread_start_us":96,"threads_started":1}
I20260809 01:52:25.874081 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.868659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.3.220.2:44929)
0809 01:52:25.869388 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.869402 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.869449 (+ 47us) server_negotiation.cc:407] Connection header received
0809 01:52:25.869567 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.869573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.869765 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.869888 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:25.870285 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.871320 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.872816 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.873388 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.873510 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.873716 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.873839 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.873930 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":117,"threads_started":1}
I20260809 01:52:25.874488 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.868431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43689 (local address 127.0.0.1:57230)
0809 01:52:25.869162 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.869176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.869266 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.869967 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.869970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.869989 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:25.870193 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.870201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.871775 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.871779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.872987 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.872998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.874081 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.874085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.874180 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.874205 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.874267 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.874343 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":61,"thread_start_us":91,"threads_started":1}
I20260809 01:52:25.874681 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.868070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:42025 (local address 127.0.0.1:43184)
0809 01:52:25.868724 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:25.868787 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:25.868914 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:25.870688 (+ 1774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:25.870691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:25.870710 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:25.870797 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.870802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.872182 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.872185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.873261 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:25.873269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.874330 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.874333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:25.874413 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.874434 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 01:52:25.874481 (+ 47us) client_negotiation.cc:240] Negotiation successful
0809 01:52:25.874555 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":80,"threads_started":1}
I20260809 01:52:25.875128 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.868873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.3.220.1:43689)
0809 01:52:25.869545 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.869552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.869575 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:52:25.869651 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.869655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.869807 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.869968 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:25.870349 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.871522 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.873136 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.873937 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.874672 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.874838 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.874891 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.874980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":75,"threads_started":1}
I20260809 01:52:25.875288 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:25.868231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.3.220.3:42025)
0809 01:52:25.870219 (+ 1988us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:25.870237 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:25.870261 (+ 24us) server_negotiation.cc:407] Connection header received
0809 01:52:25.870339 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:25.870344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:25.870544 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:25.870694 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:25.870917 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.872053 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.873400 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:25.874197 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:25.874870 (+ 673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:25.875016 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:25.875062 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 01:52:25.875143 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1889,"thread_start_us":123,"threads_started":1}
I20260809 01:52:25.877861 5791 tablet_service.cc:1467] Processing CreateTablet for tablet dbb30ce84f3947dc93d4f32c743ccbbc (DEFAULT_TABLE table=test-table [id=74fcbfb4e87a4241ae0b824699f4bf31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:25.877965 6001 tablet_service.cc:1467] Processing CreateTablet for tablet dbb30ce84f3947dc93d4f32c743ccbbc (DEFAULT_TABLE table=test-table [id=74fcbfb4e87a4241ae0b824699f4bf31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:25.877965 6204 tablet_service.cc:1467] Processing CreateTablet for tablet dbb30ce84f3947dc93d4f32c743ccbbc (DEFAULT_TABLE table=test-table [id=74fcbfb4e87a4241ae0b824699f4bf31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:25.878983 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb30ce84f3947dc93d4f32c743ccbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:25.879006 6204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb30ce84f3947dc93d4f32c743ccbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:25.879001 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb30ce84f3947dc93d4f32c743ccbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:25.885440 6332 tablet_bootstrap.cc:492] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: Bootstrap starting.
I20260809 01:52:25.886706 6333 tablet_bootstrap.cc:492] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Bootstrap starting.
I20260809 01:52:25.887640 6334 tablet_bootstrap.cc:492] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: Bootstrap starting.
I20260809 01:52:25.887982 6332 tablet_bootstrap.cc:654] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:25.888892 6333 tablet_bootstrap.cc:654] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:25.888933 6332 log.cc:824] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:25.889581 6334 tablet_bootstrap.cc:654] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:25.889804 6333 log.cc:824] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:25.890381 6334 log.cc:824] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:25.891165 6332 tablet_bootstrap.cc:492] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: No bootstrap required, opened a new log
I20260809 01:52:25.891369 6332 ts_tablet_manager.cc:1397] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:52:25.891753 6333 tablet_bootstrap.cc:492] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: No bootstrap required, opened a new log
I20260809 01:52:25.891934 6333 ts_tablet_manager.cc:1397] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:52:25.892345 6334 tablet_bootstrap.cc:492] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: No bootstrap required, opened a new log
I20260809 01:52:25.892519 6334 ts_tablet_manager.cc:1397] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 01:52:25.894830 6332 raft_consensus.cc:348] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:25.895069 6332 raft_consensus.cc:374] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:25.895135 6332 raft_consensus.cc:729] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223b5686a79e49a2ad43a3e6da8d1250, State: Initialized, Role: FOLLOWER
I20260809 01:52:25.895597 6332 consensus_queue.cc:260] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:25.895812 6333 raft_consensus.cc:348] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:25.896062 6333 raft_consensus.cc:374] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:25.896121 6333 raft_consensus.cc:729] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cdd6c5bfd74a4f9101f4e86c73894d, State: Initialized, Role: FOLLOWER
I20260809 01:52:25.896347 5887 heartbeater.cc:503] Master 127.3.220.62:34587 was elected leader, sending a full tablet report...
I20260809 01:52:25.896245 6334 raft_consensus.cc:348] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:25.896461 6334 raft_consensus.cc:374] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:25.896519 6334 raft_consensus.cc:729] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d452a92fde4d6dbf85704d9b27cbc8, State: Initialized, Role: FOLLOWER
I20260809 01:52:25.896559 6332 ts_tablet_manager.cc:1428] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:52:25.897004 6334 consensus_queue.cc:260] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:25.897066 6333 consensus_queue.cc:260] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:25.897825 6100 heartbeater.cc:503] Master 127.3.220.62:34587 was elected leader, sending a full tablet report...
I20260809 01:52:25.898051 6333 ts_tablet_manager.cc:1428] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:52:25.898082 6298 heartbeater.cc:503] Master 127.3.220.62:34587 was elected leader, sending a full tablet report...
I20260809 01:52:25.898713 6334 ts_tablet_manager.cc:1428] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:52:26.010574 6340 raft_consensus.cc:482] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:26.010964 6340 raft_consensus.cc:504] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:26.012213 6340 leader_election.cc:288] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20cdd6c5bfd74a4f9101f4e86c73894d (127.3.220.3:42025), 223b5686a79e49a2ad43a3e6da8d1250 (127.3.220.1:43689)
I20260809 01:52:26.017686 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.012663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:39417 (local address 127.3.220.1:43689)
0809 01:52:26.012844 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.012850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.012870 (+ 20us) server_negotiation.cc:407] Connection header received
0809 01:52:26.012935 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.012940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.012996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.013065 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:26.013461 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.014257 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.016223 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.016814 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.016909 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.016976 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.017440 (+ 464us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.017525 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 01:52:26.017685 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.012757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:37185 (local address 127.3.220.3:42025)
0809 01:52:26.012913 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.012917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.013518 (+ 601us) server_negotiation.cc:407] Connection header received
0809 01:52:26.013644 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.013646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.013701 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.013772 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:26.014127 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.014943 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.016161 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.016799 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.016909 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.016976 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.017440 (+ 464us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.017524 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 01:52:26.017755 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.012565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43689 (local address 127.3.220.2:39417)
0809 01:52:26.012724 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.012740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.012845 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.013139 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.013145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.013169 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:26.013315 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.013322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.014372 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.014376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.016054 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.016068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.017039 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.017043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.017246 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.017304 (+ 58us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.017383 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.017545 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":162}
I20260809 01:52:26.017982 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.012581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:42025 (local address 127.3.220.2:37185)
0809 01:52:26.013393 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.013410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.013550 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.013809 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.013812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.013831 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:26.013952 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.013958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.015149 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.015155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.016033 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.016042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.017039 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.017043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.017224 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.017301 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.017383 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.017545 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":478,"thread_start_us":203,"threads_started":1}
I20260809 01:52:26.018743 5814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb30ce84f3947dc93d4f32c743ccbbc" candidate_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223b5686a79e49a2ad43a3e6da8d1250" is_pre_election: true
I20260809 01:52:26.018777 6232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb30ce84f3947dc93d4f32c743ccbbc" candidate_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" is_pre_election: true
I20260809 01:52:26.019074 6232 raft_consensus.cc:2454] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d452a92fde4d6dbf85704d9b27cbc8 in term 0.
I20260809 01:52:26.019081 5814 raft_consensus.cc:2454] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d452a92fde4d6dbf85704d9b27cbc8 in term 0.
I20260809 01:52:26.019615 5927 leader_election.cc:302] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20cdd6c5bfd74a4f9101f4e86c73894d, 223b5686a79e49a2ad43a3e6da8d1250, f1d452a92fde4d6dbf85704d9b27cbc8; no voters:
I20260809 01:52:26.019932 6340 raft_consensus.cc:2781] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:26.020008 6340 raft_consensus.cc:482] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:26.020058 6340 raft_consensus.cc:3037] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.021186 6340 raft_consensus.cc:504] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:26.021616 6340 leader_election.cc:288] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [CANDIDATE]: Term 1 election: Requested vote from peers 20cdd6c5bfd74a4f9101f4e86c73894d (127.3.220.3:42025), 223b5686a79e49a2ad43a3e6da8d1250 (127.3.220.1:43689)
I20260809 01:52:26.021929 6232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb30ce84f3947dc93d4f32c743ccbbc" candidate_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d"
I20260809 01:52:26.022006 5814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb30ce84f3947dc93d4f32c743ccbbc" candidate_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223b5686a79e49a2ad43a3e6da8d1250"
I20260809 01:52:26.022073 6232 raft_consensus.cc:3037] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.022153 5814 raft_consensus.cc:3037] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.023236 5814 raft_consensus.cc:2454] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d452a92fde4d6dbf85704d9b27cbc8 in term 1.
I20260809 01:52:26.023391 6232 raft_consensus.cc:2454] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d452a92fde4d6dbf85704d9b27cbc8 in term 1.
I20260809 01:52:26.023586 5927 leader_election.cc:302] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223b5686a79e49a2ad43a3e6da8d1250, f1d452a92fde4d6dbf85704d9b27cbc8; no voters:
I20260809 01:52:26.023808 6340 raft_consensus.cc:2781] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:26.024008 6340 raft_consensus.cc:686] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 LEADER]: Becoming Leader. State: Replica: f1d452a92fde4d6dbf85704d9b27cbc8, State: Running, Role: LEADER
I20260809 01:52:26.024363 6340 consensus_queue.cc:237] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } }
I20260809 01:52:26.027948 5599 catalog_manager.cc:5495] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1d452a92fde4d6dbf85704d9b27cbc8 (127.3.220.2). New cstate: current_term: 1 leader_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } health_report { overall_health: HEALTHY } } }
W20260809 01:52:26.061782 6300 tablet.cc:2369] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:26.068602 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.064964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.3.220.2:44929)
0809 01:52:26.065096 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.065099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.065113 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:26.065322 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.065324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.065381 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.065474 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:26.065751 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.066377 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.067215 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.067440 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.067517 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.067838 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:26.068263 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:26.068350 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.068440 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.068489 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 01:52:26.074894 5814 raft_consensus.cc:1261] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Refusing update from remote peer f1d452a92fde4d6dbf85704d9b27cbc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:26.074913 6232 raft_consensus.cc:1261] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Refusing update from remote peer f1d452a92fde4d6dbf85704d9b27cbc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:26.075687 6345 consensus_queue.cc:1035] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:26.075887 6340 consensus_queue.cc:1035] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:26.084900 6350 mvcc.cc:205] Tried to move back new op lower bound from 7316440457515212800 to 7316440457318780928. Current Snapshot: MvccSnapshot[applied={T|T < 7316440457515212800}]
I20260809 01:52:26.089212 6349 mvcc.cc:205] Tried to move back new op lower bound from 7316440457515212800 to 7316440457318780928. Current Snapshot: MvccSnapshot[applied={T|T < 7316440457515212800 or (T in {7316440457515212800})}]
I20260809 01:52:26.097707 5599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:26.097904 5599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:26.100740 5599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e: Sending DeleteTablet for 3 replicas of tablet dbb30ce84f3947dc93d4f32c743ccbbc
I20260809 01:52:26.101725 6204 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb30ce84f3947dc93d4f32c743ccbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:26 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:43184
I20260809 01:52:26.102069 6001 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb30ce84f3947dc93d4f32c743ccbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:26 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:47236
I20260809 01:52:26.102284 5791 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb30ce84f3947dc93d4f32c743ccbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:26 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:57230
I20260809 01:52:26.102761 6364 tablet_replica.cc:330] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: stopping tablet replica
I20260809 01:52:26.102886 6363 tablet_replica.cc:330] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: stopping tablet replica
I20260809 01:52:26.102882 5599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:52:26.103216 6363 raft_consensus.cc:2229] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 01:52:26.103312 5599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:26.103437 6364 raft_consensus.cc:2229] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:52:26.103590 6363 raft_consensus.cc:2258] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:26.103940 6364 raft_consensus.cc:2258] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:26.105701 6364 ts_tablet_manager.cc:1905] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:26.108276 5791 tablet_service.cc:1467] Processing CreateTablet for tablet a9980cb88be84f33918beae31f938c8c (DEFAULT_TABLE table=test-table [id=7afd174229c0495b9e6b7e1e5b4a7095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:26.108563 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9980cb88be84f33918beae31f938c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:26.108741 6001 tablet_service.cc:1467] Processing CreateTablet for tablet a9980cb88be84f33918beae31f938c8c (DEFAULT_TABLE table=test-table [id=7afd174229c0495b9e6b7e1e5b4a7095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:26.109000 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9980cb88be84f33918beae31f938c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:26.110291 6364 ts_tablet_manager.cc:1918] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:26.110385 6364 log.cc:1196] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/dbb30ce84f3947dc93d4f32c743ccbbc
I20260809 01:52:26.110724 6364 ts_tablet_manager.cc:1939] T dbb30ce84f3947dc93d4f32c743ccbbc P f1d452a92fde4d6dbf85704d9b27cbc8: Deleting consensus metadata
I20260809 01:52:26.111060 6332 tablet_bootstrap.cc:492] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: Bootstrap starting.
I20260809 01:52:26.111493 6334 tablet_bootstrap.cc:492] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: Bootstrap starting.
I20260809 01:52:26.112084 6332 tablet_bootstrap.cc:654] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:26.112478 6334 tablet_bootstrap.cc:654] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:26.113358 6332 tablet_bootstrap.cc:492] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: No bootstrap required, opened a new log
I20260809 01:52:26.113437 6332 ts_tablet_manager.cc:1397] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:26.113487 6204 tablet_service.cc:1467] Processing CreateTablet for tablet a9980cb88be84f33918beae31f938c8c (DEFAULT_TABLE table=test-table [id=7afd174229c0495b9e6b7e1e5b4a7095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:26.113795 6204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9980cb88be84f33918beae31f938c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:26.114043 6332 raft_consensus.cc:348] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.114152 6332 raft_consensus.cc:374] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:26.114183 6332 raft_consensus.cc:729] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223b5686a79e49a2ad43a3e6da8d1250, State: Initialized, Role: FOLLOWER
I20260809 01:52:26.114317 6332 consensus_queue.cc:260] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.114718 5581 catalog_manager.cc:4841] TS f1d452a92fde4d6dbf85704d9b27cbc8 (127.3.220.2:44929): tablet dbb30ce84f3947dc93d4f32c743ccbbc (table test-table [id=74fcbfb4e87a4241ae0b824699f4bf31]) successfully deleted
I20260809 01:52:26.115316 6332 ts_tablet_manager.cc:1428] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:26.116304 6333 tablet_bootstrap.cc:492] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: Bootstrap starting.
W20260809 01:52:26.116586 5888 tablet.cc:2369] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:26.117304 6333 tablet_bootstrap.cc:654] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:26.118372 6363 ts_tablet_manager.cc:1905] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:26.120692 6363 ts_tablet_manager.cc:1918] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:26.120779 6363 log.cc:1196] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/dbb30ce84f3947dc93d4f32c743ccbbc
I20260809 01:52:26.121095 6363 ts_tablet_manager.cc:1939] T dbb30ce84f3947dc93d4f32c743ccbbc P 20cdd6c5bfd74a4f9101f4e86c73894d: Deleting consensus metadata
I20260809 01:52:26.122366 6333 tablet_bootstrap.cc:492] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: No bootstrap required, opened a new log
I20260809 01:52:26.122426 6366 tablet_replica.cc:330] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: stopping tablet replica
I20260809 01:52:26.122450 6333 ts_tablet_manager.cc:1397] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260809 01:52:26.122447 5580 catalog_manager.cc:4841] TS 20cdd6c5bfd74a4f9101f4e86c73894d (127.3.220.3:42025): tablet dbb30ce84f3947dc93d4f32c743ccbbc (table test-table [id=74fcbfb4e87a4241ae0b824699f4bf31]) successfully deleted
I20260809 01:52:26.122766 6366 raft_consensus.cc:2229] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:26.122920 6333 raft_consensus.cc:348] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.123018 6366 raft_consensus.cc:2258] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:26.123040 6333 raft_consensus.cc:374] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:26.123072 6333 raft_consensus.cc:729] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cdd6c5bfd74a4f9101f4e86c73894d, State: Initialized, Role: FOLLOWER
I20260809 01:52:26.123206 6333 consensus_queue.cc:260] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.123450 6333 ts_tablet_manager.cc:1428] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:26.123660 6334 tablet_bootstrap.cc:492] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: No bootstrap required, opened a new log
I20260809 01:52:26.123768 6334 ts_tablet_manager.cc:1397] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260809 01:52:26.124239 6334 raft_consensus.cc:348] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.124359 6334 raft_consensus.cc:374] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:26.124393 6334 raft_consensus.cc:729] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d452a92fde4d6dbf85704d9b27cbc8, State: Initialized, Role: FOLLOWER
I20260809 01:52:26.124516 6334 consensus_queue.cc:260] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.124596 6366 ts_tablet_manager.cc:1905] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:26.124791 6334 ts_tablet_manager.cc:1428] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:26.127794 6366 ts_tablet_manager.cc:1918] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:26.127877 6366 log.cc:1196] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/dbb30ce84f3947dc93d4f32c743ccbbc
I20260809 01:52:26.128237 6366 ts_tablet_manager.cc:1939] T dbb30ce84f3947dc93d4f32c743ccbbc P 223b5686a79e49a2ad43a3e6da8d1250: Deleting consensus metadata
I20260809 01:52:26.129271 5577 catalog_manager.cc:4841] TS 223b5686a79e49a2ad43a3e6da8d1250 (127.3.220.1:43689): tablet dbb30ce84f3947dc93d4f32c743ccbbc (table test-table [id=74fcbfb4e87a4241ae0b824699f4bf31]) successfully deleted
I20260809 01:52:26.212024 6342 raft_consensus.cc:482] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:26.212189 6342 raft_consensus.cc:504] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.213274 6342 leader_election.cc:288] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223b5686a79e49a2ad43a3e6da8d1250 (127.3.220.1:43689), f1d452a92fde4d6dbf85704d9b27cbc8 (127.3.220.2:44929)
I20260809 01:52:26.217917 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.213485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43689 (local address 127.3.220.3:47061)
0809 01:52:26.213717 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.213734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.213829 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.214173 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.214176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.214193 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:26.214317 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.214325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.215433 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.215437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.216698 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.216726 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.217393 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.217397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.217523 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.217596 (+ 73us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.217675 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.217750 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":42}
I20260809 01:52:26.217996 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.213776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:47061 (local address 127.3.220.1:43689)
0809 01:52:26.213914 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.213921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.213937 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:52:26.213992 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.213995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.214044 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.214115 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:26.214470 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.215313 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.216889 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.217271 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.217388 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.217451 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.217779 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.217864 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:52:26.218119 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.214008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:44395 (local address 127.3.220.2:44929)
0809 01:52:26.214138 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.214143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.214156 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:52:26.214208 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.214213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.214265 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.214365 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:26.214682 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.215180 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.216863 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.217468 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.217629 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.217706 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.217904 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.217988 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 01:52:26.218436 5814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9980cb88be84f33918beae31f938c8c" candidate_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223b5686a79e49a2ad43a3e6da8d1250" is_pre_election: true
I20260809 01:52:26.218605 5814 raft_consensus.cc:2454] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cdd6c5bfd74a4f9101f4e86c73894d in term 0.
I20260809 01:52:26.218716 6373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.213580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:44929 (local address 127.3.220.3:44395)
0809 01:52:26.213930 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.213945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.214053 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.214385 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.214388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.214405 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:26.214529 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.214534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.215319 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.215321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.216698 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.216726 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.217648 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.217654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.217761 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.217781 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.218466 (+ 685us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.218544 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":6,"thread_start_us":88,"threads_started":1}
I20260809 01:52:26.219013 6140 leader_election.cc:302] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cdd6c5bfd74a4f9101f4e86c73894d, 223b5686a79e49a2ad43a3e6da8d1250; no voters:
I20260809 01:52:26.219300 6342 raft_consensus.cc:2781] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:26.219273 6028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9980cb88be84f33918beae31f938c8c" candidate_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" is_pre_election: true
I20260809 01:52:26.219359 6342 raft_consensus.cc:482] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:26.219399 6342 raft_consensus.cc:3037] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.219508 6028 raft_consensus.cc:2454] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cdd6c5bfd74a4f9101f4e86c73894d in term 0.
I20260809 01:52:26.220587 6342 raft_consensus.cc:504] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.220953 6342 leader_election.cc:288] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [CANDIDATE]: Term 1 election: Requested vote from peers 223b5686a79e49a2ad43a3e6da8d1250 (127.3.220.1:43689), f1d452a92fde4d6dbf85704d9b27cbc8 (127.3.220.2:44929)
I20260809 01:52:26.221262 6028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9980cb88be84f33918beae31f938c8c" candidate_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8"
I20260809 01:52:26.221355 6028 raft_consensus.cc:3037] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.221311 5814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9980cb88be84f33918beae31f938c8c" candidate_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223b5686a79e49a2ad43a3e6da8d1250"
I20260809 01:52:26.221427 5814 raft_consensus.cc:3037] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.222371 6028 raft_consensus.cc:2454] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cdd6c5bfd74a4f9101f4e86c73894d in term 1.
I20260809 01:52:26.222649 6143 leader_election.cc:302] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cdd6c5bfd74a4f9101f4e86c73894d, f1d452a92fde4d6dbf85704d9b27cbc8; no voters:
I20260809 01:52:26.222718 5814 raft_consensus.cc:2454] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cdd6c5bfd74a4f9101f4e86c73894d in term 1.
I20260809 01:52:26.222832 6342 raft_consensus.cc:2781] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:26.223067 6342 raft_consensus.cc:686] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 LEADER]: Becoming Leader. State: Replica: 20cdd6c5bfd74a4f9101f4e86c73894d, State: Running, Role: LEADER
I20260809 01:52:26.223344 6342 consensus_queue.cc:237] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } }
I20260809 01:52:26.225812 5599 catalog_manager.cc:5495] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d reported cstate change: term changed from 0 to 1, leader changed from <none> to 20cdd6c5bfd74a4f9101f4e86c73894d (127.3.220.3). New cstate: current_term: 1 leader_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } health_report { overall_health: HEALTHY } } }
Aug 09 01:52:26 dist-test-slave-gkw7 krb5kdc[5534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:26.272984 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.263560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:34587 (local address 127.0.0.1:58774)
0809 01:52:26.263977 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.264006 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.264133 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.264463 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.264466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.264912 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:26.265027 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.265035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.265850 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.265853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.266477 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.266486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.266913 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.266916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.266989 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.267714 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:26.267729 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:26.269402 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:26.271273 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:26.271283 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:26.271288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:26.271553 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:26.271841 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:26.271844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:26.271845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:26.271940 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:26.272294 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:26.272299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:26.272464 (+ 165us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.272642 (+ 178us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.272789 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":189,"threads_started":1}
I20260809 01:52:26.272985 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.263608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.3.220.62:34587)
0809 01:52:26.263760 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.263764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.264087 (+ 323us) server_negotiation.cc:407] Connection header received
0809 01:52:26.264244 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.264250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.264309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.264409 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:26.265160 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.265747 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.266584 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.266795 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.266870 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.269605 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:26.269608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:26.269610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:26.269631 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:26.271165 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:26.271671 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:26.271673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:26.271674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:26.271756 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:26.272025 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:26.272027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:26.272028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:26.272193 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:26.272270 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.272691 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.272847 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":61}
I20260809 01:52:26.277130 5599 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:58774:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 01:52:26.278000 5599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:26.283293 6204 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3a631c5fc44085a5710bb54ff2ce06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8b7c90c00e24fcdaf75efc61105658e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:52:26.283277 6001 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3a631c5fc44085a5710bb54ff2ce06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8b7c90c00e24fcdaf75efc61105658e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:52:26.283681 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3a631c5fc44085a5710bb54ff2ce06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:26.283694 6204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3a631c5fc44085a5710bb54ff2ce06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:26.283531 5791 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3a631c5fc44085a5710bb54ff2ce06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8b7c90c00e24fcdaf75efc61105658e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:52:26.283924 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3a631c5fc44085a5710bb54ff2ce06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:26.286564 6332 tablet_bootstrap.cc:492] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: Bootstrap starting.
I20260809 01:52:26.286573 6333 tablet_bootstrap.cc:492] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: Bootstrap starting.
I20260809 01:52:26.287426 6333 tablet_bootstrap.cc:654] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:26.287449 6332 tablet_bootstrap.cc:654] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:26.288527 6334 tablet_bootstrap.cc:492] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: Bootstrap starting.
I20260809 01:52:26.288668 6332 tablet_bootstrap.cc:492] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: No bootstrap required, opened a new log
I20260809 01:52:26.288760 6332 ts_tablet_manager.cc:1397] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:26.289211 6332 raft_consensus.cc:348] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
I20260809 01:52:26.289295 6333 tablet_bootstrap.cc:492] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: No bootstrap required, opened a new log
I20260809 01:52:26.289335 6332 raft_consensus.cc:374] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:26.289382 6332 raft_consensus.cc:729] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223b5686a79e49a2ad43a3e6da8d1250, State: Initialized, Role: FOLLOWER
I20260809 01:52:26.289398 6333 ts_tablet_manager.cc:1397] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:26.289445 6334 tablet_bootstrap.cc:654] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:26.289570 6332 consensus_queue.cc:260] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
I20260809 01:52:26.289896 6332 ts_tablet_manager.cc:1428] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:26.289903 6338 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.289960 6333 raft_consensus.cc:348] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
I20260809 01:52:26.290076 6333 raft_consensus.cc:374] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:26.290103 6338 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:26.290109 6333 raft_consensus.cc:729] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cdd6c5bfd74a4f9101f4e86c73894d, State: Initialized, Role: FOLLOWER
I20260809 01:52:26.290243 6333 consensus_queue.cc:260] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
I20260809 01:52:26.290555 6333 ts_tablet_manager.cc:1428] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:26.290545 6342 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.290684 6342 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:26.291425 6334 tablet_bootstrap.cc:492] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: No bootstrap required, opened a new log
I20260809 01:52:26.291495 6334 ts_tablet_manager.cc:1397] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 01:52:26.291970 6334 raft_consensus.cc:348] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
I20260809 01:52:26.292083 6334 raft_consensus.cc:374] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:26.292124 6334 raft_consensus.cc:729] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d452a92fde4d6dbf85704d9b27cbc8, State: Initialized, Role: FOLLOWER
I20260809 01:52:26.292235 6334 consensus_queue.cc:260] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
I20260809 01:52:26.292537 6334 ts_tablet_manager.cc:1428] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:26.292485 6345 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.292603 6345 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:26.336936 6338 raft_consensus.cc:482] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:26.337066 6338 raft_consensus.cc:504] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
W20260809 01:52:26.337489 6101 tablet.cc:2369] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:26.338470 6338 leader_election.cc:288] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1d452a92fde4d6dbf85704d9b27cbc8 (127.3.220.2:44929), 20cdd6c5bfd74a4f9101f4e86c73894d (127.3.220.3:42025)
I20260809 01:52:26.342724 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.338637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:44929 (local address 127.3.220.1:34893)
0809 01:52:26.339051 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.339064 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.339157 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.339461 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.339463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.339483 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:26.339633 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.339640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.340547 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.340553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.341391 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.341399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.342226 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.342231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.342341 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.342368 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.342445 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.342512 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":123,"threads_started":1}
I20260809 01:52:26.342731 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.338765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:34893 (local address 127.3.220.2:44929)
0809 01:52:26.338910 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.338917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.339149 (+ 232us) server_negotiation.cc:407] Connection header received
0809 01:52:26.339288 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.339292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.339348 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.339436 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:26.339757 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.340399 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.341525 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.342090 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.342183 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.342299 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.342480 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.342562 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:52:26.342968 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.338869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:47997 (local address 127.3.220.3:42025)
0809 01:52:26.338996 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.339000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.339239 (+ 239us) server_negotiation.cc:407] Connection header received
0809 01:52:26.339467 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.339470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.339517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.339604 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:26.339926 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.340402 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.341782 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.342351 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.342463 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.342519 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.342749 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.342837 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:52:26.342973 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.338765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:42025 (local address 127.3.220.1:47997)
0809 01:52:26.339135 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.339151 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.339326 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.339636 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.339638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.339654 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:26.339805 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.339812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.340520 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.340527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.341631 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.341642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.342492 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.342496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.342604 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.342630 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.342716 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.342798 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":49,"thread_start_us":152,"threads_started":1}
I20260809 01:52:26.343263 6028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3a631c5fc44085a5710bb54ff2ce06" candidate_uuid: "223b5686a79e49a2ad43a3e6da8d1250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" is_pre_election: true
I20260809 01:52:26.343405 6028 raft_consensus.cc:2454] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 223b5686a79e49a2ad43a3e6da8d1250 in term 0.
I20260809 01:52:26.343546 6232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3a631c5fc44085a5710bb54ff2ce06" candidate_uuid: "223b5686a79e49a2ad43a3e6da8d1250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" is_pre_election: true
I20260809 01:52:26.343698 6232 raft_consensus.cc:2454] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 223b5686a79e49a2ad43a3e6da8d1250 in term 0.
I20260809 01:52:26.343815 5726 leader_election.cc:302] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223b5686a79e49a2ad43a3e6da8d1250, f1d452a92fde4d6dbf85704d9b27cbc8; no voters:
I20260809 01:52:26.344143 6338 raft_consensus.cc:2781] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:26.344205 6338 raft_consensus.cc:482] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:26.344262 6338 raft_consensus.cc:3037] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.345283 6338 raft_consensus.cc:504] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
I20260809 01:52:26.345670 6338 leader_election.cc:288] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [CANDIDATE]: Term 1 election: Requested vote from peers f1d452a92fde4d6dbf85704d9b27cbc8 (127.3.220.2:44929), 20cdd6c5bfd74a4f9101f4e86c73894d (127.3.220.3:42025)
I20260809 01:52:26.346002 6232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3a631c5fc44085a5710bb54ff2ce06" candidate_uuid: "223b5686a79e49a2ad43a3e6da8d1250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d"
I20260809 01:52:26.346002 6028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3a631c5fc44085a5710bb54ff2ce06" candidate_uuid: "223b5686a79e49a2ad43a3e6da8d1250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8"
I20260809 01:52:26.346122 6232 raft_consensus.cc:3037] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.346122 6028 raft_consensus.cc:3037] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:26.347340 6232 raft_consensus.cc:2454] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 223b5686a79e49a2ad43a3e6da8d1250 in term 1.
I20260809 01:52:26.347357 6028 raft_consensus.cc:2454] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 223b5686a79e49a2ad43a3e6da8d1250 in term 1.
I20260809 01:52:26.347651 5725 leader_election.cc:302] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cdd6c5bfd74a4f9101f4e86c73894d, 223b5686a79e49a2ad43a3e6da8d1250; no voters:
I20260809 01:52:26.347848 6338 raft_consensus.cc:2781] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:26.348105 6338 raft_consensus.cc:686] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 LEADER]: Becoming Leader. State: Replica: 223b5686a79e49a2ad43a3e6da8d1250, State: Running, Role: LEADER
I20260809 01:52:26.348364 6338 consensus_queue.cc:237] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } }
I20260809 01:52:26.350104 6395 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: TxnStatusTablet state changed. Reason: New leader 223b5686a79e49a2ad43a3e6da8d1250. Latest consensus state: current_term: 1 leader_uuid: "223b5686a79e49a2ad43a3e6da8d1250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.350216 6395 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:52:26.350570 6401 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 01:52:26.350793 6401 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 01:52:26.350638 5599 catalog_manager.cc:5495] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 reported cstate change: term changed from 0 to 1, leader changed from <none> to 223b5686a79e49a2ad43a3e6da8d1250 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "223b5686a79e49a2ad43a3e6da8d1250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } health_report { overall_health: HEALTHY } } }
I20260809 01:52:26.372013 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.368063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43689 (local address 127.0.0.1:40662)
0809 01:52:26.368305 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.368317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.368446 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.368795 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.368798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.368807 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:26.368919 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.368925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.369834 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.369837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.370442 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.370450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.370920 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.370924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.371014 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.371067 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:26.371717 (+ 650us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:26.371742 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.371807 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.371893 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260809 01:52:26.372013 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.368124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.3.220.1:43689)
0809 01:52:26.368255 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.368260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.368384 (+ 124us) server_negotiation.cc:407] Connection header received
0809 01:52:26.368593 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.368596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.368661 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.368784 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:26.369036 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.369660 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.370565 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.370808 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.370916 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.371221 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:26.371620 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:26.371743 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.371829 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.371884 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 01:52:26.375085 6232 raft_consensus.cc:1261] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Refusing update from remote peer 223b5686a79e49a2ad43a3e6da8d1250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:26.375366 6028 raft_consensus.cc:1261] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Refusing update from remote peer 223b5686a79e49a2ad43a3e6da8d1250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:26.375597 6338 consensus_queue.cc:1035] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:26.376049 6338 consensus_queue.cc:1035] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:26.377295 6342 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: TxnStatusTablet state changed. Reason: New leader 223b5686a79e49a2ad43a3e6da8d1250. Latest consensus state: current_term: 1 leader_uuid: "223b5686a79e49a2ad43a3e6da8d1250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.377411 6342 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:26.379300 6345 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: TxnStatusTablet state changed. Reason: New leader 223b5686a79e49a2ad43a3e6da8d1250. Latest consensus state: current_term: 1 leader_uuid: "223b5686a79e49a2ad43a3e6da8d1250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.379427 6345 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:26.380926 6338 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "223b5686a79e49a2ad43a3e6da8d1250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.381038 6338 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:52:26.381990 6345 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "223b5686a79e49a2ad43a3e6da8d1250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.382109 6345 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P f1d452a92fde4d6dbf85704d9b27cbc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:26.389074 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.385425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.3.220.3:42025)
0809 01:52:26.385577 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.385582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.385658 (+ 76us) server_negotiation.cc:407] Connection header received
0809 01:52:26.385845 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.385847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.385909 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.386015 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:26.386271 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.386800 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.387637 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.387869 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.387971 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.388240 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:26.388684 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:26.388814 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.388884 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.388935 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 01:52:26.390755 6338 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "223b5686a79e49a2ad43a3e6da8d1250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.390882 6338 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P 223b5686a79e49a2ad43a3e6da8d1250: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:52:26.391599 6342 tablet_replica.cc:439] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "223b5686a79e49a2ad43a3e6da8d1250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 } } peers { permanent_uuid: "20cdd6c5bfd74a4f9101f4e86c73894d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 42025 } } peers { permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 } } }
I20260809 01:52:26.391710 6342 tablet_replica.cc:442] T 1c3a631c5fc44085a5710bb54ff2ce06 P 20cdd6c5bfd74a4f9101f4e86c73894d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:26.401854 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.397662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43689 (local address 127.3.220.3:52221)
0809 01:52:26.397968 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.397984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.398089 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.398513 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.398516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.398522 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:26.398631 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.398635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.399664 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.399667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.400387 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.400396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.401126 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.401129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.401203 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.401260 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:26.401587 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:26.401593 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.401651 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.401713 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":136,"threads_started":1}
I20260809 01:52:26.401877 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.397732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:52221 (local address 127.3.220.1:43689)
0809 01:52:26.397865 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.397870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.398043 (+ 173us) server_negotiation.cc:407] Connection header received
0809 01:52:26.398188 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.398190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.398238 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.398343 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:26.398737 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.399535 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.400522 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.400990 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.401113 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.401400 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:26.401484 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:26.401581 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.401684 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.401737 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:52:26.408779 6028 raft_consensus.cc:1261] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Refusing update from remote peer 20cdd6c5bfd74a4f9101f4e86c73894d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:26.408812 5814 raft_consensus.cc:1261] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Refusing update from remote peer 20cdd6c5bfd74a4f9101f4e86c73894d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:26.409282 6375 consensus_queue.cc:1035] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [LEADER]: Connected to new peer: Peer: permanent_uuid: "223b5686a79e49a2ad43a3e6da8d1250" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:26.409754 6342 consensus_queue.cc:1035] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1d452a92fde4d6dbf85704d9b27cbc8" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:26.432760 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.428801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:42025 (local address 127.3.220.1:57499)
0809 01:52:26.429151 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:26.429168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:26.429295 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:26.429507 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:26.429510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:26.429518 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:26.429695 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.429701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.430378 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.430383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.431374 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:26.431384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.432079 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.432082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:26.432166 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.432212 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:26.432507 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:26.432512 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 01:52:26.432562 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 01:52:26.432623 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":119,"threads_started":1}
I20260809 01:52:26.432762 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.428845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:57499 (local address 127.3.220.3:42025)
0809 01:52:26.429104 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.429108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.429224 (+ 116us) server_negotiation.cc:407] Connection header received
0809 01:52:26.429372 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.429374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.429412 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.429473 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:26.429782 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.430267 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.431611 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.431972 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.432078 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.432346 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:26.432427 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:26.432498 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.432572 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.432624 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260809 01:52:26.461619 5596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:26.461818 5596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:26.464267 5596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65dafd2a63094de897507a2fd0292e9e: Sending DeleteTablet for 3 replicas of tablet a9980cb88be84f33918beae31f938c8c
I20260809 01:52:26.464815 5791 tablet_service.cc:1514] Processing DeleteTablet for tablet a9980cb88be84f33918beae31f938c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:26 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:57230
I20260809 01:52:26.465003 6366 tablet_replica.cc:330] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: stopping tablet replica
I20260809 01:52:26.465116 6366 raft_consensus.cc:2229] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:26.465183 6001 tablet_service.cc:1514] Processing DeleteTablet for tablet a9980cb88be84f33918beae31f938c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:26 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:47236
I20260809 01:52:26.465224 6366 raft_consensus.cc:2258] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:26.465360 6364 tablet_replica.cc:330] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: stopping tablet replica
I20260809 01:52:26.465473 6364 raft_consensus.cc:2229] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:26.465623 6364 raft_consensus.cc:2258] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:26.466012 6204 tablet_service.cc:1514] Processing DeleteTablet for tablet a9980cb88be84f33918beae31f938c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:26 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:43184
I20260809 01:52:26.466179 6363 tablet_replica.cc:330] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: stopping tablet replica
I20260809 01:52:26.466277 6363 raft_consensus.cc:2229] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:52:26.466409 6366 ts_tablet_manager.cc:1905] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:26.466547 6363 raft_consensus.cc:2258] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:26.467478 6364 ts_tablet_manager.cc:1905] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:26.469748 6366 ts_tablet_manager.cc:1918] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:26.469879 6366 log.cc:1196] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/a9980cb88be84f33918beae31f938c8c
I20260809 01:52:26.470239 6366 ts_tablet_manager.cc:1939] T a9980cb88be84f33918beae31f938c8c P 223b5686a79e49a2ad43a3e6da8d1250: Deleting consensus metadata
I20260809 01:52:26.470310 6364 ts_tablet_manager.cc:1918] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:26.470399 6364 log.cc:1196] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/a9980cb88be84f33918beae31f938c8c
I20260809 01:52:26.470687 6364 ts_tablet_manager.cc:1939] T a9980cb88be84f33918beae31f938c8c P f1d452a92fde4d6dbf85704d9b27cbc8: Deleting consensus metadata
I20260809 01:52:26.471356 5581 catalog_manager.cc:4841] TS f1d452a92fde4d6dbf85704d9b27cbc8 (127.3.220.2:44929): tablet a9980cb88be84f33918beae31f938c8c (table test-table [id=7afd174229c0495b9e6b7e1e5b4a7095]) successfully deleted
I20260809 01:52:26.471545 5577 catalog_manager.cc:4841] TS 223b5686a79e49a2ad43a3e6da8d1250 (127.3.220.1:43689): tablet a9980cb88be84f33918beae31f938c8c (table test-table [id=7afd174229c0495b9e6b7e1e5b4a7095]) successfully deleted
I20260809 01:52:26.473304 6363 ts_tablet_manager.cc:1905] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:26.476018 6363 ts_tablet_manager.cc:1918] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:26.476109 6363 log.cc:1196] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/a9980cb88be84f33918beae31f938c8c
I20260809 01:52:26.476397 6363 ts_tablet_manager.cc:1939] T a9980cb88be84f33918beae31f938c8c P 20cdd6c5bfd74a4f9101f4e86c73894d: Deleting consensus metadata
I20260809 01:52:26.477161 5580 catalog_manager.cc:4841] TS 20cdd6c5bfd74a4f9101f4e86c73894d (127.3.220.3:42025): tablet a9980cb88be84f33918beae31f938c8c (table test-table [id=7afd174229c0495b9e6b7e1e5b4a7095]) successfully deleted
Aug 09 01:52:26 dist-test-slave-gkw7 krb5kdc[5534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:26.490463 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.473200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.3.220.1:43689)
0809 01:52:26.477080 (+ 3880us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.477085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.477431 (+ 346us) server_negotiation.cc:407] Connection header received
0809 01:52:26.477590 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.477593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.477644 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.477730 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:26.478616 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.479450 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.480817 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.481161 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.481276 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.485187 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:26.485197 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:26.485213 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:26.485251 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:26.488267 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:26.488824 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:26.488831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:26.488833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:26.488944 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:26.489261 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:26.489264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:26.489266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:26.489653 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:26.489982 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.490091 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.490275 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3793}
W20260809 01:52:26.491070 5872 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40678
I20260809 01:52:26.502707 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:26.493122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.3.220.62:34587)
0809 01:52:26.493256 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:26.493260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:26.493788 (+ 528us) server_negotiation.cc:407] Connection header received
0809 01:52:26.493902 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:26.493905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:26.493960 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:26.494046 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:26.494854 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.495684 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.496799 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:26.497163 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:26.497268 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:26.499114 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:26.499118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:26.499121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:26.499150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:26.500859 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:26.501330 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:26.501334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:26.501336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:26.501422 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:26.501777 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:26.501780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:26.501782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:26.502012 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:26.502141 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:26.502422 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 01:52:26.502540 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 01:52:26.503291 5596 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58788
I20260809 01:52:26.504714 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 5702
I20260809 01:52:26.527556 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 5900
I20260809 01:52:26.546324 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 6110
I20260809 01:52:26.559427 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 5554
2026-08-09T01:52:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3158 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:26 dist-test-slave-gkw7 krb5kdc[6437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:26 dist-test-slave-gkw7 krb5kdc[6437](info): set up 2 sockets
Aug 09 01:52:26 dist-test-slave-gkw7 krb5kdc[6437](info): commencing operation
krb5kdc: starting...
W20260809 01:52:28.052495 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.464s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:28 dist-test-slave-gkw7 krb5kdc[6437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:28.180459 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44239
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:28.294241 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:28.294561 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:28.294670 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:28.300182 6457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:28.300256 6457 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 01:52:28.300282 6457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:28.300315 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:28.300351 6457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:28.300380 6457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:28.307542 6457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:44239
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:28.308737 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:28.310015 6457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:28.319032 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:28.319178 6457 server_base.cc:1034] running on GCE node
W20260809 01:52:28.319029 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:28.319341 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:28.319962 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:28.321090 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:28.322295 6457 hybrid_clock.cc:648] HybridClock initialized: now 1786240348322264 us; error 43 us; skew 500 ppm
Aug 09 01:52:28 dist-test-slave-gkw7 krb5kdc[6437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:28.330470 6457 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:28.331419 6457 webserver.cc:460] Webserver started at http://127.3.220.62:34613/ using document root <none> and password file <none>
I20260809 01:52:28.332006 6457 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:28.332098 6457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:28.332355 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:28.334554 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "5687742b4bb745a6b1d1bbe48ecfee31"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "bf1f6d64f7ebed39170e709cd00ba3e7"
server_key_iv: "80418c69b0063773b92359b7a5391e83"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.335139 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "5687742b4bb745a6b1d1bbe48ecfee31"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "bf1f6d64f7ebed39170e709cd00ba3e7"
server_key_iv: "80418c69b0063773b92359b7a5391e83"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.339066 6457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:52:28.341753 6477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:28.342931 6457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 01:52:28.343072 6457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "5687742b4bb745a6b1d1bbe48ecfee31"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "bf1f6d64f7ebed39170e709cd00ba3e7"
server_key_iv: "80418c69b0063773b92359b7a5391e83"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.343186 6457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:28.354086 6457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:28.356724 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:28.356918 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:28.365016 6457 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:44239
I20260809 01:52:28.365041 6576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:44239 every 8 connection(s)
I20260809 01:52:28.365942 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:28.366694 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 6457
I20260809 01:52:28.366822 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:28.367080 3952 external_mini_cluster.cc:1428] Setting key 9535474eddc1c7133d245ab6fa2189cd
I20260809 01:52:28.368865 6577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:28.375447 6577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: Bootstrap starting.
Aug 09 01:52:28 dist-test-slave-gkw7 krb5kdc[6437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:28.378090 6577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:28.379499 6577 log.cc:824] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:28.381825 6577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: No bootstrap required, opened a new log
I20260809 01:52:28.381942 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.368423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.3.220.62:44239)
0809 01:52:28.368841 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:28.368862 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:28.368896 (+ 34us) server_negotiation.cc:407] Connection header received
0809 01:52:28.369618 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:28.369637 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:28.369987 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:28.370255 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:28.371179 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.372158 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.373325 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.373846 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.373963 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.376717 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:28.376724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:28.376739 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:28.376770 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:28.378846 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.379271 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.379279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.379286 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.379435 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.379710 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.379713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.379715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.380177 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:28.380350 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:28.380676 (+ 326us) server_negotiation.cc:299] Negotiation successful
0809 01:52:28.380964 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":130,"threads_started":1}
I20260809 01:52:28.385147 6577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5687742b4bb745a6b1d1bbe48ecfee31" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44239 } }
I20260809 01:52:28.385365 6577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:28.385399 6577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5687742b4bb745a6b1d1bbe48ecfee31, State: Initialized, Role: FOLLOWER
I20260809 01:52:28.385810 6577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5687742b4bb745a6b1d1bbe48ecfee31" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44239 } }
I20260809 01:52:28.385931 6577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:28.385974 6577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:28.386044 6577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:28.386865 6577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5687742b4bb745a6b1d1bbe48ecfee31" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44239 } }
I20260809 01:52:28.387147 6577 leader_election.cc:302] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5687742b4bb745a6b1d1bbe48ecfee31; no voters:
I20260809 01:52:28.387398 6577 leader_election.cc:288] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:28.387530 6584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:28.387768 6584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [term 1 LEADER]: Becoming Leader. State: Replica: 5687742b4bb745a6b1d1bbe48ecfee31, State: Running, Role: LEADER
I20260809 01:52:28.388082 6584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5687742b4bb745a6b1d1bbe48ecfee31" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44239 } }
I20260809 01:52:28.388633 6577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:28.389703 6588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5687742b4bb745a6b1d1bbe48ecfee31. Latest consensus state: current_term: 1 leader_uuid: "5687742b4bb745a6b1d1bbe48ecfee31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5687742b4bb745a6b1d1bbe48ecfee31" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44239 } } }
I20260809 01:52:28.389854 6588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:28.390241 6586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5687742b4bb745a6b1d1bbe48ecfee31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5687742b4bb745a6b1d1bbe48ecfee31" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44239 } } }
I20260809 01:52:28.390401 6586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:28.391139 6599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:28.393720 6599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:28.399171 6599 catalog_manager.cc:1347] Generated new cluster ID: 1266e4331db6431da757cdaee9759f11
I20260809 01:52:28.399235 6599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:28.414252 6599 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:28.415560 6599 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:28.425415 6599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: Generated new TSK 0
I20260809 01:52:28.426249 6599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:28.462548 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44239
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:28.576905 6615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:28.577165 6615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:28.577212 6615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:28.581173 6615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:28.581224 6615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:28.581301 6615 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 01:52:28.581333 6615 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:52:28.581391 6615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:28.586517 6615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:28.587554 6615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:28.588687 6615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:28.598338 6622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:28.598327 6624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:28.598327 6621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:28.598744 6615 server_base.cc:1034] running on GCE node
I20260809 01:52:28.599145 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:28.599764 6615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:28.600960 6615 hybrid_clock.cc:648] HybridClock initialized: now 1786240348600940 us; error 30 us; skew 500 ppm
Aug 09 01:52:28 dist-test-slave-gkw7 krb5kdc[6437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:28.603808 6615 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:28.604713 6615 webserver.cc:460] Webserver started at http://127.3.220.1:38649/ using document root <none> and password file <none>
I20260809 01:52:28.605291 6615 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:28.605384 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:28.605679 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:28.607887 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "70e3de3b7cb5a276c19612f3e620d477"
server_key_iv: "da1181c6c2bc7ddfe3ce340390b4245e"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.608530 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "70e3de3b7cb5a276c19612f3e620d477"
server_key_iv: "da1181c6c2bc7ddfe3ce340390b4245e"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.612425 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260809 01:52:28.615116 6638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:28.616374 6615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:28.616514 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "70e3de3b7cb5a276c19612f3e620d477"
server_key_iv: "da1181c6c2bc7ddfe3ce340390b4245e"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.616631 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:28.634012 6615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:28.636880 6615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:28.637094 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:28.638624 6615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:28.638697 6615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:28.638783 6615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:28.638823 6615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 01:52:28 dist-test-slave-gkw7 krb5kdc[6437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:28.650350 6615 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:45233
I20260809 01:52:28.650655 6800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:45233 every 8 connection(s)
I20260809 01:52:28.651491 6615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:28.653414 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.639351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:40809 (local address 127.3.220.62:44239)
0809 01:52:28.639483 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:28.639486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:28.641293 (+ 1807us) server_negotiation.cc:407] Connection header received
0809 01:52:28.642599 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:28.642603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:28.642659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:28.642711 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:28.644158 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.644807 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.646134 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.646555 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.646710 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.649522 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:28.649543 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:28.649546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:28.649580 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:28.651279 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.651820 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.651825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.651827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.651931 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.652322 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.652327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.652328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.652512 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:28.652622 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:28.653105 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 01:52:28.653242 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 01:52:28.654137 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.640520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44239 (local address 127.3.220.1:40809)
0809 01:52:28.641084 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:28.641171 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:28.642354 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:28.643083 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:28.643091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:28.643537 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:28.644005 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:28.644018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.644939 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.644942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:28.645990 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:28.646000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.646758 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.646762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:28.646992 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.647650 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:28.647666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:28.649345 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:28.651389 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:28.651393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:28.651404 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:28.651709 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:28.652048 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:28.652051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:28.652054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:28.652232 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:28.652633 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:28.652641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:28.652871 (+ 230us) client_negotiation.cc:769] Sending connection context
0809 01:52:28.653062 (+ 191us) client_negotiation.cc:240] Negotiation successful
0809 01:52:28.653359 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":130,"threads_started":1}
I20260809 01:52:28.655433 6801 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44239
I20260809 01:52:28.655591 6801 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:28.656067 6801 heartbeater.cc:511] Master 127.3.220.62:44239 requested a full tablet report, sending...
I20260809 01:52:28.657753 6501 ts_manager.cc:194] Registered new tserver with Master: fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233)
I20260809 01:52:28.658874 6501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:40809
I20260809 01:52:28.660017 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 6615
I20260809 01:52:28.660144 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:28.660444 3952 external_mini_cluster.cc:1428] Setting key 5ac9f411569f885cebbc38d9cc0afe5d
I20260809 01:52:28.666528 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.657602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:50773 (local address 127.3.220.62:44239)
0809 01:52:28.657718 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:28.657721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:28.659612 (+ 1891us) server_negotiation.cc:407] Connection header received
0809 01:52:28.659726 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:28.659728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:28.659765 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:28.659859 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:28.660377 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.661024 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.661974 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.662206 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.662315 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.663438 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:28.663442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:28.663445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:28.663468 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:28.664864 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.665290 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.665294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.665296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.665376 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.665715 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.665719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.665721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.665893 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:28.666004 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:28.666271 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 01:52:28.666378 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 01:52:28.666538 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.657496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44239 (local address 127.3.220.1:50773)
0809 01:52:28.659527 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:28.659540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:28.659619 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:28.659881 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:28.659884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:28.660091 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:28.660265 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:28.660272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.661156 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.661159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:28.661862 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:28.661871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.662320 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.662323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:28.662425 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.662933 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:28.662944 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:28.663306 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:28.664987 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:28.664990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:28.664992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:28.665191 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:28.665478 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:28.665481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:28.665483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:28.665621 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:28.665994 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:28.665997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:28.666093 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 01:52:28.666201 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 01:52:28.666341 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1947,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:28.689242 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44239
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:28.803644 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:28.803902 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:28.803951 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:28.808138 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:28.808204 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:28.808250 6820 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 01:52:28.808300 6820 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:52:28.808359 6820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:28.813649 6820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:28.814924 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:28.816155 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:28.826265 6833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:28.826280 6827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:28.826439 6820 server_base.cc:1034] running on GCE node
W20260809 01:52:28.826517 6828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:28.827023 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:28.827677 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:28.828847 6820 hybrid_clock.cc:648] HybridClock initialized: now 1786240348828836 us; error 42 us; skew 500 ppm
Aug 09 01:52:28 dist-test-slave-gkw7 krb5kdc[6437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:28.831656 6820 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:28.832564 6820 webserver.cc:460] Webserver started at http://127.3.220.2:34757/ using document root <none> and password file <none>
I20260809 01:52:28.833112 6820 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:28.833171 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:28.833343 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:28.835503 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "82d983f31c244226b419087158729032"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "de095b0f89157968a786d21a22e398d1"
server_key_iv: "8fe73b60ade5510bd218e555e7327f5b"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.836035 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "82d983f31c244226b419087158729032"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "de095b0f89157968a786d21a22e398d1"
server_key_iv: "8fe73b60ade5510bd218e555e7327f5b"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.839910 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:52:28.842621 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:28.843916 6820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:28.844050 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "82d983f31c244226b419087158729032"
format_stamp: "Formatted at 2026-08-09 01:52:28 on dist-test-slave-gkw7"
server_key: "de095b0f89157968a786d21a22e398d1"
server_key_iv: "8fe73b60ade5510bd218e555e7327f5b"
server_key_version: "encryptionkey@0"
I20260809 01:52:28.844167 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:28.869342 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:28.872394 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:28.872548 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:28.874094 6820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:28.874171 6820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:28.874238 6820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:28.874298 6820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:28.885172 6820 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:38169
I20260809 01:52:28.885253 7001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:38169 every 8 connection(s)
Aug 09 01:52:28 dist-test-slave-gkw7 krb5kdc[6437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:28.886272 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:28.889974 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.874774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:50177 (local address 127.3.220.62:44239)
0809 01:52:28.874966 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:28.874973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:28.876484 (+ 1511us) server_negotiation.cc:407] Connection header received
0809 01:52:28.877996 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:28.878000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:28.878080 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:28.878216 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:28.879929 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.880665 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.882512 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.882733 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.882819 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.885987 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:28.885990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:28.885993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:28.886023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:28.887742 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.888260 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.888263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.888265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.888360 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.888759 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.888762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.888764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.888953 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:28.889090 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:28.889649 (+ 559us) server_negotiation.cc:299] Negotiation successful
0809 01:52:28.889801 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 01:52:28.890877 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.875739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44239 (local address 127.3.220.2:50177)
0809 01:52:28.876312 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:28.876370 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:28.877722 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:28.878397 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:28.878409 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:28.878959 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:28.879732 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:28.879757 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.880781 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.880787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:28.882369 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:28.882384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.882904 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.882909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:28.883148 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.883901 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:28.883930 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:28.885797 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:28.887867 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:28.887872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:28.887887 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:28.888146 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:28.888456 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:28.888459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:28.888461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:28.888624 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:28.889086 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:28.889092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:28.889339 (+ 247us) client_negotiation.cc:769] Sending connection context
0809 01:52:28.889592 (+ 253us) client_negotiation.cc:240] Negotiation successful
0809 01:52:28.889928 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":149,"threads_started":1}
I20260809 01:52:28.892524 7002 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44239
I20260809 01:52:28.892730 7002 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:28.893399 7002 heartbeater.cc:511] Master 127.3.220.62:44239 requested a full tablet report, sending...
I20260809 01:52:28.894693 6502 ts_manager.cc:194] Registered new tserver with Master: 82d983f31c244226b419087158729032 (127.3.220.2:38169)
I20260809 01:52:28.895440 6502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:50177
I20260809 01:52:28.897838 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 6820
I20260809 01:52:28.897940 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:28.898248 3952 external_mini_cluster.cc:1428] Setting key f4237125a33f53428dacf83008c9b2fb
I20260809 01:52:28.905207 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.895952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44239 (local address 127.3.220.2:54587)
0809 01:52:28.896266 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:28.896278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:28.896392 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:28.896640 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:28.896643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:28.896854 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:28.896946 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:28.896952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.898079 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.898083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:28.898992 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:28.899020 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.899472 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.899476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:28.899569 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.900194 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:28.900209 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:28.900639 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:28.903229 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:28.903235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:28.903239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:28.903565 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:28.903913 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:28.903919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:28.903923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:28.904017 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:28.904605 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:28.904610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:28.904738 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 01:52:28.904847 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 01:52:28.904997 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":87,"threads_started":1}
I20260809 01:52:28.905200 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:28.895992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:54587 (local address 127.3.220.62:44239)
0809 01:52:28.896103 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:28.896107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:28.896352 (+ 245us) server_negotiation.cc:407] Connection header received
0809 01:52:28.896469 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:28.896473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:28.896516 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:28.896591 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:28.897062 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.897914 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.899135 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:28.899358 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:28.899469 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:28.900786 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:28.900790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:28.900792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:28.900820 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:28.903097 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.903692 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.903698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.903702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.903794 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:28.904120 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:28.904126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:28.904130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:28.904433 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:28.904515 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:28.904921 (+ 406us) server_negotiation.cc:299] Negotiation successful
0809 01:52:28.905040 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:28.928762 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44239
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:29.042150 7013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:29.042416 7013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:29.042497 7013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:29.046540 7013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:29.046607 7013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:29.046674 7013 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 01:52:29.046712 7013 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:52:29.046799 7013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:29.052246 7013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:29.053344 7013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:29.054472 7013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:29.064667 7019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:29.064771 7013 server_base.cc:1034] running on GCE node
W20260809 01:52:29.064778 7024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:29.064929 7021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:29.065348 7013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:29.066061 7013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:29.067302 7013 hybrid_clock.cc:648] HybridClock initialized: now 1786240349067266 us; error 28 us; skew 500 ppm
Aug 09 01:52:29 dist-test-slave-gkw7 krb5kdc[6437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:29.070209 7013 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:29.071012 7013 webserver.cc:460] Webserver started at http://127.3.220.3:41811/ using document root <none> and password file <none>
I20260809 01:52:29.071537 7013 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:29.071593 7013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:29.071755 7013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:29.073788 7013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "167d92587818462b8f08b39a8f3806bc"
format_stamp: "Formatted at 2026-08-09 01:52:29 on dist-test-slave-gkw7"
server_key: "9db6a6d40d881758d5d58d27b2413471"
server_key_iv: "1fed4b0915b485f8ea87f884ec9e6107"
server_key_version: "encryptionkey@0"
I20260809 01:52:29.074303 7013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "167d92587818462b8f08b39a8f3806bc"
format_stamp: "Formatted at 2026-08-09 01:52:29 on dist-test-slave-gkw7"
server_key: "9db6a6d40d881758d5d58d27b2413471"
server_key_iv: "1fed4b0915b485f8ea87f884ec9e6107"
server_key_version: "encryptionkey@0"
I20260809 01:52:29.077988 7013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:29.080554 7033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:29.081920 7013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:29.082018 7013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "167d92587818462b8f08b39a8f3806bc"
format_stamp: "Formatted at 2026-08-09 01:52:29 on dist-test-slave-gkw7"
server_key: "9db6a6d40d881758d5d58d27b2413471"
server_key_iv: "1fed4b0915b485f8ea87f884ec9e6107"
server_key_version: "encryptionkey@0"
I20260809 01:52:29.082106 7013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:29.097121 7013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:29.100303 7013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:29.100455 7013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:29.101827 7013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:29.101922 7013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:29.102000 7013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:29.102046 7013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 01:52:29 dist-test-slave-gkw7 krb5kdc[6437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:29.113216 7013 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:33567
I20260809 01:52:29.113278 7201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:33567 every 8 connection(s)
I20260809 01:52:29.114224 7013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:29.115531 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 7013
I20260809 01:52:29.115660 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:29.115985 3952 external_mini_cluster.cc:1428] Setting key b79c8cfe27a23d72ffffa70d986b1e5b
I20260809 01:52:29.116146 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.102653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:50671 (local address 127.3.220.62:44239)
0809 01:52:29.102786 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.102790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.103794 (+ 1004us) server_negotiation.cc:407] Connection header received
0809 01:52:29.104864 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.104868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.104931 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.105040 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:29.106408 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.106996 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.108151 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.108496 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.108610 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.111989 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:29.111993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:29.112002 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:29.112028 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:29.113758 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:29.114271 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:29.114276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:29.114279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:29.114396 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:29.114854 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:29.114857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:29.114858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:29.115099 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:29.115204 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.115855 (+ 651us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.115967 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:52:29.117094 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.103183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44239 (local address 127.3.220.3:50671)
0809 01:52:29.103637 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.103677 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.104639 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.105207 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.105218 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.105761 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:29.106253 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.106267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.107125 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.107128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.108018 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.108027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.108663 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.108667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.108836 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.109450 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:29.109469 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:29.111820 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:29.113875 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:29.113881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:29.113896 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:29.114159 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:29.114516 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:29.114519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:29.114521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:29.114722 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:29.115234 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:29.115239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:29.115565 (+ 326us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.115796 (+ 231us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.116149 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260809 01:52:29.118378 7202 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44239
I20260809 01:52:29.118538 7202 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:29.119035 7202 heartbeater.cc:511] Master 127.3.220.62:44239 requested a full tablet report, sending...
I20260809 01:52:29.120229 6502 ts_manager.cc:194] Registered new tserver with Master: 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567)
I20260809 01:52:29.120826 6502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:50671
I20260809 01:52:29.127905 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.121324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44239 (local address 127.3.220.3:57111)
0809 01:52:29.121611 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.121627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.121726 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.121996 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.121999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.122212 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:29.122353 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.122361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.123175 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.123177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.123720 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.123728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.124191 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.124193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.124280 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.124815 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:29.124831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:29.125145 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:29.126696 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:29.126699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:29.126700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:29.126887 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:29.127119 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:29.127121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:29.127123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:29.127195 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:29.127513 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:29.127515 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:29.127586 (+ 71us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.127666 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.127771 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260809 01:52:29.127933 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.121387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:57111 (local address 127.3.220.62:44239)
0809 01:52:29.121524 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.121573 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.121693 (+ 120us) server_negotiation.cc:407] Connection header received
0809 01:52:29.121814 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.121816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.121873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.121958 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:29.122476 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.123037 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.123836 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.124079 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.124180 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.125283 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:29.125293 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:29.125295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:29.125320 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:29.126600 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:29.126970 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:29.126973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:29.126974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:29.127044 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:29.127277 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:29.127280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:29.127281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:29.127435 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:29.127512 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.127736 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.127827 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:52:29.130213 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 01:52:29 dist-test-slave-gkw7 krb5kdc[6437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 01:52:29 dist-test-slave-gkw7 krb5kdc[6437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:29.156874 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.148657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.3.220.62:44239)
0809 01:52:29.148809 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.148813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.148825 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:52:29.148866 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.148868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.148906 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.148971 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:29.149769 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.150283 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.151067 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.151274 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.151347 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.153816 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:29.153820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:29.153822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:29.153845 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:29.155252 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:29.155645 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:29.155665 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:29.155667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:29.155757 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:29.155981 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:29.155983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:29.155984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:29.156155 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:29.156323 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.156510 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.156684 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 01:52:29.159893 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:52:29.162928 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:29.177608 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.170947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:33567 (local address 127.0.0.1:43322)
0809 01:52:29.171385 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.171410 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.171549 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.172930 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.172933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.172953 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:29.173049 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.173054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.174456 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.174460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.176090 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.176097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.177071 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.177074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.177221 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.177242 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.177323 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.177378 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":340,"spinlock_wait_cycles":5248,"thread_start_us":96,"threads_started":1}
I20260809 01:52:29.177629 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.171691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.3.220.3:33567)
0809 01:52:29.172487 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.172499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.172522 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:52:29.172594 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.172602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.172786 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.172935 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:29.173169 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.174327 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.176207 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.176958 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.177064 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.177218 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.177358 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.177446 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":138,"threads_started":1}
I20260809 01:52:29.177755 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.171850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.3.220.2:38169)
0809 01:52:29.172296 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.172309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.172796 (+ 487us) server_negotiation.cc:407] Connection header received
0809 01:52:29.172880 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.172888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.173053 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.173187 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:29.173608 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.174356 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.176022 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.176652 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.176778 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.176954 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.177473 (+ 519us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.177604 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":107,"threads_started":1}
I20260809 01:52:29.177619 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.170918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:45233 (local address 127.0.0.1:46126)
0809 01:52:29.171358 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.171410 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.171549 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.172361 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.172368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.172396 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:29.172541 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.172551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.173735 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.173738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.175801 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.175814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.177055 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.177059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.177260 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.177283 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.177359 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.177431 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":68,"spinlock_wait_cycles":7040,"thread_start_us":62,"threads_started":1}
I20260809 01:52:29.177387 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.171621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38169 (local address 127.0.0.1:41006)
0809 01:52:29.172591 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.172604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.172691 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.173341 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.173344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.173362 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:29.173446 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.173452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.174503 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.174506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.175892 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.175901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.176778 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.176782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.176884 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.176920 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.177042 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.177205 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":152,"thread_start_us":441,"threads_started":1}
I20260809 01:52:29.178177 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.171033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.3.220.1:45233)
0809 01:52:29.171858 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.171866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.171892 (+ 26us) server_negotiation.cc:407] Connection header received
0809 01:52:29.171970 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.171975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.172193 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.172351 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:29.172685 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.173614 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.175947 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.176726 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.176847 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.176987 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.177687 (+ 700us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.178053 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":89,"threads_started":1}
I20260809 01:52:29.180786 6709 tablet_service.cc:1467] Processing CreateTablet for tablet 0c47fdd7aa144dd886eb55ea4e456565 (DEFAULT_TABLE table=test-table [id=11ec85e281cd4aca95cf65a339404e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:29.180845 6915 tablet_service.cc:1467] Processing CreateTablet for tablet 0c47fdd7aa144dd886eb55ea4e456565 (DEFAULT_TABLE table=test-table [id=11ec85e281cd4aca95cf65a339404e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:29.180845 7111 tablet_service.cc:1467] Processing CreateTablet for tablet 0c47fdd7aa144dd886eb55ea4e456565 (DEFAULT_TABLE table=test-table [id=11ec85e281cd4aca95cf65a339404e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:29.181869 6709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c47fdd7aa144dd886eb55ea4e456565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.181885 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c47fdd7aa144dd886eb55ea4e456565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.181877 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c47fdd7aa144dd886eb55ea4e456565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.187794 7237 tablet_bootstrap.cc:492] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: Bootstrap starting.
I20260809 01:52:29.189590 7238 tablet_bootstrap.cc:492] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: Bootstrap starting.
I20260809 01:52:29.190442 7239 tablet_bootstrap.cc:492] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: Bootstrap starting.
I20260809 01:52:29.190631 7237 tablet_bootstrap.cc:654] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.191519 7237 log.cc:824] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:29.192588 7239 tablet_bootstrap.cc:654] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.193240 7238 tablet_bootstrap.cc:654] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.193449 7237 tablet_bootstrap.cc:492] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: No bootstrap required, opened a new log
I20260809 01:52:29.193470 7239 log.cc:824] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:29.193681 7237 ts_tablet_manager.cc:1397] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:52:29.194067 7238 log.cc:824] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:29.195597 7239 tablet_bootstrap.cc:492] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: No bootstrap required, opened a new log
I20260809 01:52:29.195850 7239 ts_tablet_manager.cc:1397] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 01:52:29.196029 7238 tablet_bootstrap.cc:492] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: No bootstrap required, opened a new log
I20260809 01:52:29.196208 7238 ts_tablet_manager.cc:1397] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 01:52:29.198071 7237 raft_consensus.cc:348] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.198294 7237 raft_consensus.cc:374] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.198380 7237 raft_consensus.cc:729] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc5a5bbf0d542f1b2341066f8ea36f6, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.198863 7237 consensus_queue.cc:260] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.199715 7237 ts_tablet_manager.cc:1428] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260809 01:52:29.200096 6801 heartbeater.cc:503] Master 127.3.220.62:44239 was elected leader, sending a full tablet report...
I20260809 01:52:29.199992 7238 raft_consensus.cc:348] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.200253 7238 raft_consensus.cc:374] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.200313 7238 raft_consensus.cc:729] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167d92587818462b8f08b39a8f3806bc, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.200732 7238 consensus_queue.cc:260] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.201406 7202 heartbeater.cc:503] Master 127.3.220.62:44239 was elected leader, sending a full tablet report...
I20260809 01:52:29.201737 7238 ts_tablet_manager.cc:1428] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260809 01:52:29.201131 7239 raft_consensus.cc:348] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.201965 7239 raft_consensus.cc:374] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.202025 7239 raft_consensus.cc:729] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d983f31c244226b419087158729032, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.202430 7239 consensus_queue.cc:260] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.203092 7244 raft_consensus.cc:482] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:29.203284 7244 raft_consensus.cc:504] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.203447 7239 ts_tablet_manager.cc:1428] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260809 01:52:29.203850 7002 heartbeater.cc:503] Master 127.3.220.62:44239 was elected leader, sending a full tablet report...
I20260809 01:52:29.205174 7244 leader_election.cc:288] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567), 82d983f31c244226b419087158729032 (127.3.220.2:38169)
I20260809 01:52:29.210078 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.205676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:44733 (local address 127.3.220.3:33567)
0809 01:52:29.205846 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.205851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.205865 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:29.205901 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.205905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.205954 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.206056 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:29.206323 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.207045 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.208653 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.209273 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.209390 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.209450 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.209888 (+ 438us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.209960 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 01:52:29.210078 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.205035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:41025 (local address 127.3.220.2:38169)
0809 01:52:29.205153 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.205159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.205344 (+ 185us) server_negotiation.cc:407] Connection header received
0809 01:52:29.205580 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.205585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.205636 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.205689 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:29.206431 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.206930 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.208653 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.209311 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.209403 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.209450 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.209888 (+ 438us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.209960 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:52:29.210202 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.204949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38169 (local address 127.3.220.1:41025)
0809 01:52:29.205257 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.205272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.205424 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.206185 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.206190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.206204 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:29.206313 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.206322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.207046 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.207051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.208518 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.208530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.209423 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.209427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.209686 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.209781 (+ 95us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.209860 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.210009 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":286,"thread_start_us":111,"threads_started":1}
I20260809 01:52:29.210260 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.205266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:33567 (local address 127.3.220.1:44733)
0809 01:52:29.205578 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.205593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.205701 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.206069 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.206073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.206090 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:29.206222 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.206228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.207163 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.207166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.208518 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.208530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.209423 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.209427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.209688 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.209779 (+ 91us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.209860 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.210009 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":218,"thread_start_us":113,"threads_started":1}
I20260809 01:52:29.210785 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c47fdd7aa144dd886eb55ea4e456565" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167d92587818462b8f08b39a8f3806bc" is_pre_election: true
I20260809 01:52:29.210796 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c47fdd7aa144dd886eb55ea4e456565" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d983f31c244226b419087158729032" is_pre_election: true
I20260809 01:52:29.211079 7139 raft_consensus.cc:2454] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 0.
I20260809 01:52:29.211079 6942 raft_consensus.cc:2454] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 0.
I20260809 01:52:29.211539 6643 leader_election.cc:302] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d983f31c244226b419087158729032, fbc5a5bbf0d542f1b2341066f8ea36f6; no voters:
I20260809 01:52:29.211809 7244 raft_consensus.cc:2781] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:29.211894 7244 raft_consensus.cc:482] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:29.211938 7244 raft_consensus.cc:3037] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.212939 7244 raft_consensus.cc:504] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.213241 7244 leader_election.cc:288] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 election: Requested vote from peers 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567), 82d983f31c244226b419087158729032 (127.3.220.2:38169)
I20260809 01:52:29.213589 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c47fdd7aa144dd886eb55ea4e456565" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167d92587818462b8f08b39a8f3806bc"
I20260809 01:52:29.213589 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c47fdd7aa144dd886eb55ea4e456565" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d983f31c244226b419087158729032"
I20260809 01:52:29.213712 6942 raft_consensus.cc:3037] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.213709 7139 raft_consensus.cc:3037] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.214952 7139 raft_consensus.cc:2454] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 1.
I20260809 01:52:29.214952 6942 raft_consensus.cc:2454] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 1.
I20260809 01:52:29.215253 6642 leader_election.cc:302] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167d92587818462b8f08b39a8f3806bc, fbc5a5bbf0d542f1b2341066f8ea36f6; no voters:
I20260809 01:52:29.215408 7244 raft_consensus.cc:2781] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:29.215619 7244 raft_consensus.cc:686] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 LEADER]: Becoming Leader. State: Replica: fbc5a5bbf0d542f1b2341066f8ea36f6, State: Running, Role: LEADER
I20260809 01:52:29.215941 7244 consensus_queue.cc:237] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } }
I20260809 01:52:29.218887 6500 catalog_manager.cc:5495] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } health_report { overall_health: UNKNOWN } } }
Aug 09 01:52:29 dist-test-slave-gkw7 krb5kdc[6437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:29.255890 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.247570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44239 (local address 127.0.0.1:48978)
0809 01:52:29.247935 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.247951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.248085 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.248359 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.248362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.248857 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:29.248953 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.248960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.249771 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.249777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.250335 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.250343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.250784 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.250788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.250869 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.251489 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:29.251501 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:29.252892 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:29.254470 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:29.254479 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:29.254484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:29.254705 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:29.254963 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:29.254967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:29.254968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:29.255058 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:29.255440 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:29.255446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:29.255552 (+ 106us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.255644 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.255754 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":145,"threads_started":1}
I20260809 01:52:29.255970 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.247636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.3.220.62:44239)
0809 01:52:29.247810 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.247817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.248031 (+ 214us) server_negotiation.cc:407] Connection header received
0809 01:52:29.248206 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.248208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.248253 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.248339 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:29.249066 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.249667 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.250453 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.250678 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.250784 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.253028 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:29.253031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:29.253033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:29.253054 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:29.254371 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:29.254794 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:29.254796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:29.254798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:29.254868 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:29.255134 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:29.255136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:29.255137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:29.255315 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:29.255406 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.255740 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.255835 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 01:52:29.259791 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:48978:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 01:52:29.260538 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:29.265765 6915 tablet_service.cc:1467] Processing CreateTablet for tablet 2885e3d37bf549ff9407911b281e2683 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e151f32568f4a9c96e993df164264d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:52:29.265519 6709 tablet_service.cc:1467] Processing CreateTablet for tablet 2885e3d37bf549ff9407911b281e2683 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e151f32568f4a9c96e993df164264d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:52:29.265873 7111 tablet_service.cc:1467] Processing CreateTablet for tablet 2885e3d37bf549ff9407911b281e2683 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e151f32568f4a9c96e993df164264d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:52:29.266369 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2885e3d37bf549ff9407911b281e2683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.266371 6709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2885e3d37bf549ff9407911b281e2683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.266522 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2885e3d37bf549ff9407911b281e2683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.268919 7238 tablet_bootstrap.cc:492] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: Bootstrap starting.
I20260809 01:52:29.269397 7237 tablet_bootstrap.cc:492] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: Bootstrap starting.
I20260809 01:52:29.270056 7238 tablet_bootstrap.cc:654] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.270789 7239 tablet_bootstrap.cc:492] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: Bootstrap starting.
I20260809 01:52:29.271171 7237 tablet_bootstrap.cc:654] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.271543 7238 tablet_bootstrap.cc:492] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: No bootstrap required, opened a new log
I20260809 01:52:29.271633 7238 ts_tablet_manager.cc:1397] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:29.271867 7239 tablet_bootstrap.cc:654] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.272167 7238 raft_consensus.cc:348] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.272292 7238 raft_consensus.cc:374] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.272326 7238 raft_consensus.cc:729] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167d92587818462b8f08b39a8f3806bc, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.272470 7238 consensus_queue.cc:260] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.272760 7237 tablet_bootstrap.cc:492] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: No bootstrap required, opened a new log
I20260809 01:52:29.272766 7238 ts_tablet_manager.cc:1428] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 01:52:29.272853 7237 ts_tablet_manager.cc:1397] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:29.273370 7245 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.273586 7245 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:29.273967 7237 raft_consensus.cc:348] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.274111 7237 raft_consensus.cc:374] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.274149 7237 raft_consensus.cc:729] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc5a5bbf0d542f1b2341066f8ea36f6, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.274300 7237 consensus_queue.cc:260] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.274590 7237 ts_tablet_manager.cc:1428] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:29.274619 7244 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.274746 7244 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:29.276568 7244 raft_consensus.cc:482] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:29.276686 7244 raft_consensus.cc:504] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.277132 7244 leader_election.cc:288] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d983f31c244226b419087158729032 (127.3.220.2:38169), 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567)
I20260809 01:52:29.277604 7239 tablet_bootstrap.cc:492] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: No bootstrap required, opened a new log
I20260809 01:52:29.277704 7239 ts_tablet_manager.cc:1397] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260809 01:52:29.277699 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2885e3d37bf549ff9407911b281e2683" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d983f31c244226b419087158729032" is_pre_election: true
I20260809 01:52:29.278216 7239 raft_consensus.cc:348] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.278362 7239 raft_consensus.cc:374] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.278337 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2885e3d37bf549ff9407911b281e2683" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167d92587818462b8f08b39a8f3806bc" is_pre_election: true
I20260809 01:52:29.278404 7239 raft_consensus.cc:729] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d983f31c244226b419087158729032, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.278465 7139 raft_consensus.cc:2454] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 0.
I20260809 01:52:29.278541 7239 consensus_queue.cc:260] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.278728 6942 raft_consensus.cc:2454] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 0.
I20260809 01:52:29.278808 6642 leader_election.cc:302] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167d92587818462b8f08b39a8f3806bc, fbc5a5bbf0d542f1b2341066f8ea36f6; no voters:
I20260809 01:52:29.278987 7244 raft_consensus.cc:2781] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:29.279042 7244 raft_consensus.cc:482] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:29.279078 7244 raft_consensus.cc:3037] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.279006 7246 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.279143 7246 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:29.279706 7239 ts_tablet_manager.cc:1428] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:29.280213 7244 raft_consensus.cc:504] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.280580 7244 leader_election.cc:288] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 election: Requested vote from peers 82d983f31c244226b419087158729032 (127.3.220.2:38169), 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567)
I20260809 01:52:29.281035 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2885e3d37bf549ff9407911b281e2683" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d983f31c244226b419087158729032"
I20260809 01:52:29.281180 6942 raft_consensus.cc:3037] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.281885 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2885e3d37bf549ff9407911b281e2683" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167d92587818462b8f08b39a8f3806bc"
I20260809 01:52:29.282068 7139 raft_consensus.cc:3037] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.282315 6942 raft_consensus.cc:2454] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 1.
I20260809 01:52:29.282624 6643 leader_election.cc:302] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d983f31c244226b419087158729032, fbc5a5bbf0d542f1b2341066f8ea36f6; no voters:
I20260809 01:52:29.282853 7244 raft_consensus.cc:2781] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:29.282936 7244 raft_consensus.cc:686] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 LEADER]: Becoming Leader. State: Replica: fbc5a5bbf0d542f1b2341066f8ea36f6, State: Running, Role: LEADER
I20260809 01:52:29.283056 7244 consensus_queue.cc:237] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.283232 7139 raft_consensus.cc:2454] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 1.
I20260809 01:52:29.283815 7252 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: TxnStatusTablet state changed. Reason: New leader fbc5a5bbf0d542f1b2341066f8ea36f6. Latest consensus state: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.283912 7252 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:52:29.284775 7275 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 01:52:29.284686 6500 catalog_manager.cc:5495] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } health_report { overall_health: UNKNOWN } } }
I20260809 01:52:29.284991 7275 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 01:52:29.299667 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.295864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:45233 (local address 127.0.0.1:46136)
0809 01:52:29.296024 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.296041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.296150 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.296505 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.296508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.296516 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:29.296620 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.296626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.297379 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.297383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.298087 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.298097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.298574 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.298578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.298674 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.298728 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.299366 (+ 638us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.299384 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.299452 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.299537 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 01:52:29.299667 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.295960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.3.220.1:45233)
0809 01:52:29.296082 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.296086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.296100 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:29.296319 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.296323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.296389 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.296491 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:29.296741 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.297268 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.298211 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.298444 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.298570 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.298877 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.299267 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.299392 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.299471 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.299529 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 01:52:29.304507 6942 raft_consensus.cc:1261] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Refusing update from remote peer fbc5a5bbf0d542f1b2341066f8ea36f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:29.304648 7139 raft_consensus.cc:1261] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Refusing update from remote peer fbc5a5bbf0d542f1b2341066f8ea36f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:29.305263 7244 consensus_queue.cc:1035] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:29.305485 7252 consensus_queue.cc:1035] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:29.309367 7246 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: TxnStatusTablet state changed. Reason: New leader fbc5a5bbf0d542f1b2341066f8ea36f6. Latest consensus state: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.309500 7246 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:29.309876 7245 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: TxnStatusTablet state changed. Reason: New leader fbc5a5bbf0d542f1b2341066f8ea36f6. Latest consensus state: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.309994 7245 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:29.312573 7252 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.312716 7252 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:52:29.313602 7244 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.313725 7244 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P fbc5a5bbf0d542f1b2341066f8ea36f6: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:52:29.314975 7245 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.315014 7246 tablet_replica.cc:439] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } } }
I20260809 01:52:29.315105 7245 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P 167d92587818462b8f08b39a8f3806bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:29.315124 7246 tablet_replica.cc:442] T 2885e3d37bf549ff9407911b281e2683 P 82d983f31c244226b419087158729032: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:52:29.317118 7282 mvcc.cc:205] Tried to move back new op lower bound from 7316440470744846336 to 7316440470665056256. Current Snapshot: MvccSnapshot[applied={T|T < 7316440470744846336}]
I20260809 01:52:29.318532 7281 mvcc.cc:205] Tried to move back new op lower bound from 7316440470744846336 to 7316440470665056256. Current Snapshot: MvccSnapshot[applied={T|T < 7316440470744846336}]
I20260809 01:52:29.325624 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.321760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.3.220.1:45233)
0809 01:52:29.321912 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.321918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.321961 (+ 43us) server_negotiation.cc:407] Connection header received
0809 01:52:29.322193 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.322196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.322255 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.322346 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:29.322638 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.323212 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.324336 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.324618 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.324749 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.325082 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.325185 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.325285 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.325389 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.325444 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:52:29.337518 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.333767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:55635 (local address 127.3.220.1:45233)
0809 01:52:29.333891 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.333896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.334179 (+ 283us) server_negotiation.cc:407] Connection header received
0809 01:52:29.334356 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.334361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.334410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.334499 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:29.334776 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.335324 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.336279 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.336510 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.336638 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.336970 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.337047 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.337139 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.337268 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.337323 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":49}
I20260809 01:52:29.337568 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.333688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:45233 (local address 127.3.220.1:55635)
0809 01:52:29.334048 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.334059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.334251 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.334508 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.334510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.334517 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:29.334668 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.334673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.335443 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.335448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.336143 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.336152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.336632 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.336635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.336748 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.336791 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.337153 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.337161 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.337246 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.337330 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":23,"thread_start_us":111,"threads_started":1}
I20260809 01:52:29.344789 6942 raft_consensus.cc:1261] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Refusing update from remote peer fbc5a5bbf0d542f1b2341066f8ea36f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:29.344807 7139 raft_consensus.cc:1261] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Refusing update from remote peer fbc5a5bbf0d542f1b2341066f8ea36f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:29.345165 7252 consensus_queue.cc:1035] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:29.345342 7297 consensus_queue.cc:1035] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:29.348575 7279 mvcc.cc:205] Tried to move back new op lower bound from 7316440470912385024 to 7316440470391345152. Current Snapshot: MvccSnapshot[applied={T|T < 7316440470912385024}]
I20260809 01:52:29.400966 6502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:29.401157 6502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:29.403926 6502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: Sending DeleteTablet for 3 replicas of tablet 0c47fdd7aa144dd886eb55ea4e456565
I20260809 01:52:29.404837 7111 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c47fdd7aa144dd886eb55ea4e456565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:29 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:43322
I20260809 01:52:29.405040 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c47fdd7aa144dd886eb55ea4e456565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:29 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:41006
I20260809 01:52:29.405390 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c47fdd7aa144dd886eb55ea4e456565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:29 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:46126
I20260809 01:52:29.405625 7306 tablet_replica.cc:330] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: stopping tablet replica
I20260809 01:52:29.405994 7306 raft_consensus.cc:2229] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:29.406082 7305 tablet_replica.cc:330] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: stopping tablet replica
I20260809 01:52:29.406229 7306 raft_consensus.cc:2258] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:29.406379 7305 raft_consensus.cc:2229] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:29.406580 7305 raft_consensus.cc:2258] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:29.406814 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:52:29.406915 7308 tablet_replica.cc:330] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: stopping tablet replica
I20260809 01:52:29.407189 7308 raft_consensus.cc:2229] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 LEADER]: Raft consensus shutting down.
W20260809 01:52:29.407285 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:29.407546 7308 raft_consensus.cc:2258] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:29.408782 7306 ts_tablet_manager.cc:1905] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:29.410386 7308 ts_tablet_manager.cc:1905] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:29.413496 6915 tablet_service.cc:1467] Processing CreateTablet for tablet 1047df897da34579b13c1ec1da3a3e6e (DEFAULT_TABLE table=test-table [id=277bf40113cc4ac2b287ab98cdf533ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:29.413681 7308 ts_tablet_manager.cc:1918] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:29.413805 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1047df897da34579b13c1ec1da3a3e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.413817 7111 tablet_service.cc:1467] Processing CreateTablet for tablet 1047df897da34579b13c1ec1da3a3e6e (DEFAULT_TABLE table=test-table [id=277bf40113cc4ac2b287ab98cdf533ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:29.413870 7308 log.cc:1196] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/0c47fdd7aa144dd886eb55ea4e456565
I20260809 01:52:29.414083 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1047df897da34579b13c1ec1da3a3e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.414290 7308 ts_tablet_manager.cc:1939] T 0c47fdd7aa144dd886eb55ea4e456565 P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting consensus metadata
I20260809 01:52:29.415624 6484 catalog_manager.cc:4841] TS fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233): tablet 0c47fdd7aa144dd886eb55ea4e456565 (table test-table [id=11ec85e281cd4aca95cf65a339404e37]) successfully deleted
I20260809 01:52:29.416535 7239 tablet_bootstrap.cc:492] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: Bootstrap starting.
I20260809 01:52:29.416755 7238 tablet_bootstrap.cc:492] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: Bootstrap starting.
I20260809 01:52:29.413523 6707 tablet_service.cc:1467] Processing CreateTablet for tablet 1047df897da34579b13c1ec1da3a3e6e (DEFAULT_TABLE table=test-table [id=277bf40113cc4ac2b287ab98cdf533ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:29.417080 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1047df897da34579b13c1ec1da3a3e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:29.417605 7239 tablet_bootstrap.cc:654] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.417766 7238 tablet_bootstrap.cc:654] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.419736 7237 tablet_bootstrap.cc:492] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: Bootstrap starting.
I20260809 01:52:29.420364 7306 ts_tablet_manager.cc:1918] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:29.420528 7306 log.cc:1196] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/0c47fdd7aa144dd886eb55ea4e456565
I20260809 01:52:29.420805 7237 tablet_bootstrap.cc:654] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:29.421036 7305 ts_tablet_manager.cc:1905] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:29.421244 7306 ts_tablet_manager.cc:1939] T 0c47fdd7aa144dd886eb55ea4e456565 P 82d983f31c244226b419087158729032: Deleting consensus metadata
I20260809 01:52:29.422300 7237 tablet_bootstrap.cc:492] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: No bootstrap required, opened a new log
I20260809 01:52:29.422403 7237 ts_tablet_manager.cc:1397] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:29.422713 6484 catalog_manager.cc:4841] TS 82d983f31c244226b419087158729032 (127.3.220.2:38169): tablet 0c47fdd7aa144dd886eb55ea4e456565 (table test-table [id=11ec85e281cd4aca95cf65a339404e37]) successfully deleted
I20260809 01:52:29.422960 7237 raft_consensus.cc:348] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.423110 7237 raft_consensus.cc:374] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.423168 7237 raft_consensus.cc:729] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc5a5bbf0d542f1b2341066f8ea36f6, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.423322 7237 consensus_queue.cc:260] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.423630 7237 ts_tablet_manager.cc:1428] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260809 01:52:29.424103 7305 ts_tablet_manager.cc:1918] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:29.424237 7305 log.cc:1196] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/0c47fdd7aa144dd886eb55ea4e456565
I20260809 01:52:29.424631 7305 ts_tablet_manager.cc:1939] T 0c47fdd7aa144dd886eb55ea4e456565 P 167d92587818462b8f08b39a8f3806bc: Deleting consensus metadata
I20260809 01:52:29.424634 7239 tablet_bootstrap.cc:492] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: No bootstrap required, opened a new log
I20260809 01:52:29.424782 7239 ts_tablet_manager.cc:1397] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260809 01:52:29.425207 7239 raft_consensus.cc:348] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.425339 7239 raft_consensus.cc:374] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.425385 7239 raft_consensus.cc:729] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d983f31c244226b419087158729032, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.425542 7239 consensus_queue.cc:260] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.425874 7239 ts_tablet_manager.cc:1428] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 01:52:29.426447 6483 catalog_manager.cc:4841] TS 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567): tablet 0c47fdd7aa144dd886eb55ea4e456565 (table test-table [id=11ec85e281cd4aca95cf65a339404e37]) successfully deleted
I20260809 01:52:29.427071 7238 tablet_bootstrap.cc:492] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: No bootstrap required, opened a new log
I20260809 01:52:29.427145 7238 ts_tablet_manager.cc:1397] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260809 01:52:29.427579 7238 raft_consensus.cc:348] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.427706 7238 raft_consensus.cc:374] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:29.427754 7238 raft_consensus.cc:729] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167d92587818462b8f08b39a8f3806bc, State: Initialized, Role: FOLLOWER
I20260809 01:52:29.427888 7238 consensus_queue.cc:260] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.428228 7238 ts_tablet_manager.cc:1428] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:29.568116 7246 raft_consensus.cc:482] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:29.568341 7246 raft_consensus.cc:504] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.569659 7246 leader_election.cc:288] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233), 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567)
I20260809 01:52:29.574905 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.570404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:51937 (local address 127.3.220.3:33567)
0809 01:52:29.570540 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.570544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.570592 (+ 48us) server_negotiation.cc:407] Connection header received
0809 01:52:29.571083 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.571086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.571131 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.571221 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:29.571571 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.572071 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.573815 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.574247 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.574380 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.574455 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.574693 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.574776 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 01:52:29.575132 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.569978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:42575 (local address 127.3.220.1:45233)
0809 01:52:29.570260 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.570267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.570574 (+ 307us) server_negotiation.cc:407] Connection header received
0809 01:52:29.570812 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.570816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.570869 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.570953 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:29.571355 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.572107 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.573842 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.574432 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.574510 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.574571 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.574912 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.575005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260809 01:52:29.574952 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.569974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:33567 (local address 127.3.220.2:51937)
0809 01:52:29.570510 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.570525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.570679 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.571286 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.571292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.571316 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:29.571438 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.571444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.572269 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.572273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.573675 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.573686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.574395 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.574399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.574508 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.574535 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.574656 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.574740 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":57,"spinlock_wait_cycles":23552,"thread_start_us":120,"threads_started":1}
I20260809 01:52:29.575582 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.569907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:45233 (local address 127.3.220.2:42575)
0809 01:52:29.570468 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.570486 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.570679 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.570983 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.570985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.571001 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:29.571133 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.571139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.572279 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.572282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.573699 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.573710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.574655 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.574659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.574756 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.574779 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.575301 (+ 522us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.575392 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":121,"spinlock_wait_cycles":9728,"thread_start_us":103,"threads_started":1}
I20260809 01:52:29.575843 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1047df897da34579b13c1ec1da3a3e6e" candidate_uuid: "82d983f31c244226b419087158729032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167d92587818462b8f08b39a8f3806bc" is_pre_election: true
I20260809 01:52:29.576033 7139 raft_consensus.cc:2454] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d983f31c244226b419087158729032 in term 0.
I20260809 01:52:29.576262 6736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1047df897da34579b13c1ec1da3a3e6e" candidate_uuid: "82d983f31c244226b419087158729032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" is_pre_election: true
I20260809 01:52:29.576501 6736 raft_consensus.cc:2454] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d983f31c244226b419087158729032 in term 0.
I20260809 01:52:29.576602 6850 leader_election.cc:302] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167d92587818462b8f08b39a8f3806bc, 82d983f31c244226b419087158729032; no voters:
I20260809 01:52:29.576907 7246 raft_consensus.cc:2781] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:29.576975 7246 raft_consensus.cc:482] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:29.577013 7246 raft_consensus.cc:3037] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.578356 7246 raft_consensus.cc:504] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.578732 7246 leader_election.cc:288] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [CANDIDATE]: Term 1 election: Requested vote from peers fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233), 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567)
I20260809 01:52:29.579092 6736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1047df897da34579b13c1ec1da3a3e6e" candidate_uuid: "82d983f31c244226b419087158729032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6"
I20260809 01:52:29.579109 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1047df897da34579b13c1ec1da3a3e6e" candidate_uuid: "82d983f31c244226b419087158729032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167d92587818462b8f08b39a8f3806bc"
I20260809 01:52:29.579226 7139 raft_consensus.cc:3037] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.579242 6736 raft_consensus.cc:3037] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:29.580308 6736 raft_consensus.cc:2454] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d983f31c244226b419087158729032 in term 1.
I20260809 01:52:29.580642 6851 leader_election.cc:302] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d983f31c244226b419087158729032, fbc5a5bbf0d542f1b2341066f8ea36f6; no voters:
I20260809 01:52:29.580843 7139 raft_consensus.cc:2454] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d983f31c244226b419087158729032 in term 1.
I20260809 01:52:29.581049 7246 raft_consensus.cc:2781] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:29.581245 7246 raft_consensus.cc:686] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 1 LEADER]: Becoming Leader. State: Replica: 82d983f31c244226b419087158729032, State: Running, Role: LEADER
I20260809 01:52:29.581509 7246 consensus_queue.cc:237] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:29.584450 6500 catalog_manager.cc:5495] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d983f31c244226b419087158729032 (127.3.220.2). New cstate: current_term: 1 leader_uuid: "82d983f31c244226b419087158729032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } health_report { overall_health: UNKNOWN } } }
W20260809 01:52:29.616283 7203 tablet.cc:2369] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:29.624457 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.620171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.3.220.2:38169)
0809 01:52:29.620293 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.620298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.620314 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:52:29.620506 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.620509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.620573 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.620699 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:29.620903 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.621642 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.622521 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.622893 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.622997 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.623295 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.624012 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.624140 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.624241 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.624298 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:52:29.636819 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.632644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:45233 (local address 127.3.220.2:52337)
0809 01:52:29.633020 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.633033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.633156 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.633446 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.633449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.633457 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:29.633624 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.633635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.634379 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.634384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.635542 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.635553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.636017 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.636021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.636118 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.636172 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.636485 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.636495 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.636560 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.636645 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
I20260809 01:52:29.636783 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.632768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:52337 (local address 127.3.220.1:45233)
0809 01:52:29.632910 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.632917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.633104 (+ 187us) server_negotiation.cc:407] Connection header received
0809 01:52:29.633288 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.633291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.633344 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.633424 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:29.633745 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.634269 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.635672 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.635905 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.636016 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.636323 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.636403 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.636481 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.636570 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.636626 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 01:52:29.638849 7003 tablet.cc:2369] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:29.643497 7139 raft_consensus.cc:1261] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Refusing update from remote peer 82d983f31c244226b419087158729032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:29.643699 6736 raft_consensus.cc:1261] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Refusing update from remote peer 82d983f31c244226b419087158729032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:29.644011 7246 consensus_queue.cc:1035] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:29.644222 7318 consensus_queue.cc:1035] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:29.664345 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.660487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:48047 (local address 127.3.220.2:38169)
0809 01:52:29.660593 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:29.660598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:29.660829 (+ 231us) server_negotiation.cc:407] Connection header received
0809 01:52:29.660900 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:29.660903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:29.660956 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:29.661030 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:29.661296 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.662104 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.663300 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.663545 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.663653 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.663908 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.663975 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.664062 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:29.664156 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 01:52:29.664212 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 01:52:29.664348 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:29.660422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38169 (local address 127.3.220.1:48047)
0809 01:52:29.660610 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:29.660622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:29.660738 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:29.661040 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:29.661043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:29.661049 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:29.661162 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.661180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.662220 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.662223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.663170 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:29.663180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:29.663666 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:29.663670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:29.663763 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:29.663795 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:29.664070 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:29.664079 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 01:52:29.664146 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 01:52:29.664241 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
W20260809 01:52:29.665146 7282 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 01:52:31.223182 6500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:31.223336 6500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:31.225725 6500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: Sending DeleteTablet for 3 replicas of tablet 1047df897da34579b13c1ec1da3a3e6e
I20260809 01:52:31.226327 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 1047df897da34579b13c1ec1da3a3e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:31 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:41006
I20260809 01:52:31.226329 7111 tablet_service.cc:1514] Processing DeleteTablet for tablet 1047df897da34579b13c1ec1da3a3e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:31 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:43322
I20260809 01:52:31.226342 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet 1047df897da34579b13c1ec1da3a3e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:31 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:46126
I20260809 01:52:31.226681 7361 tablet_replica.cc:330] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: stopping tablet replica
I20260809 01:52:31.226818 7361 raft_consensus.cc:2229] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:31.226943 7361 raft_consensus.cc:2258] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:31.227691 7361 ts_tablet_manager.cc:1905] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:31.230125 7360 tablet_replica.cc:330] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: stopping tablet replica
I20260809 01:52:31.230347 7359 tablet_replica.cc:330] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: stopping tablet replica
I20260809 01:52:31.230499 7359 raft_consensus.cc:2229] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:52:31.230815 7359 raft_consensus.cc:2258] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:31.231040 7361 ts_tablet_manager.cc:1918] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 01:52:31.231036 7360 raft_consensus.cc:2229] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:31.231132 7361 log.cc:1196] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/1047df897da34579b13c1ec1da3a3e6e
I20260809 01:52:31.231171 7360 raft_consensus.cc:2258] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:31.231456 7361 ts_tablet_manager.cc:1939] T 1047df897da34579b13c1ec1da3a3e6e P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting consensus metadata
I20260809 01:52:31.231722 7359 ts_tablet_manager.cc:1905] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:31.231766 7360 ts_tablet_manager.cc:1905] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:31.232205 6484 catalog_manager.cc:4841] TS fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233): tablet 1047df897da34579b13c1ec1da3a3e6e (table test-table [id=277bf40113cc4ac2b287ab98cdf533ac]) successfully deleted
I20260809 01:52:31.233979 7360 ts_tablet_manager.cc:1918] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 01:52:31.234071 7360 log.cc:1196] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/1047df897da34579b13c1ec1da3a3e6e
I20260809 01:52:31.234346 7360 ts_tablet_manager.cc:1939] T 1047df897da34579b13c1ec1da3a3e6e P 167d92587818462b8f08b39a8f3806bc: Deleting consensus metadata
I20260809 01:52:31.234998 6483 catalog_manager.cc:4841] TS 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567): tablet 1047df897da34579b13c1ec1da3a3e6e (table test-table [id=277bf40113cc4ac2b287ab98cdf533ac]) successfully deleted
I20260809 01:52:31.235014 7359 ts_tablet_manager.cc:1918] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 01:52:31.235105 7359 log.cc:1196] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/1047df897da34579b13c1ec1da3a3e6e
I20260809 01:52:31.235422 7359 ts_tablet_manager.cc:1939] T 1047df897da34579b13c1ec1da3a3e6e P 82d983f31c244226b419087158729032: Deleting consensus metadata
I20260809 01:52:31.236184 6484 catalog_manager.cc:4841] TS 82d983f31c244226b419087158729032 (127.3.220.2:38169): tablet 1047df897da34579b13c1ec1da3a3e6e (table test-table [id=277bf40113cc4ac2b287ab98cdf533ac]) successfully deleted
Aug 09 01:52:31 dist-test-slave-gkw7 krb5kdc[6437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240351, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 01:52:31 dist-test-slave-gkw7 krb5kdc[6437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240351, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:31.258708 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.249502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.3.220.62:44239)
0809 01:52:31.249835 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:31.249846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:31.249865 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:52:31.250040 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:31.250043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:31.250097 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:31.250193 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:31.250919 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.251503 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.252376 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.252622 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.252739 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.255265 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:31.255269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:31.255272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:31.255303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:31.256961 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:31.257390 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:31.257393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:31.257395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:31.257483 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:31.257809 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:31.257812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:31.257814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:31.258006 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:31.258147 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:31.258395 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 01:52:31.258526 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260809 01:52:31.261364 6500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:52:31.261855 6500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:31.266987 6707 tablet_service.cc:1467] Processing CreateTablet for tablet 4cea676c551a4599bb7291ef9182d765 (DEFAULT_TABLE table=test-table [id=e6716c49f8ed4d52a6f1fd5bd630457b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:31.267040 6915 tablet_service.cc:1467] Processing CreateTablet for tablet 4cea676c551a4599bb7291ef9182d765 (DEFAULT_TABLE table=test-table [id=e6716c49f8ed4d52a6f1fd5bd630457b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:31.267269 7111 tablet_service.cc:1467] Processing CreateTablet for tablet 4cea676c551a4599bb7291ef9182d765 (DEFAULT_TABLE table=test-table [id=e6716c49f8ed4d52a6f1fd5bd630457b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:31.267352 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cea676c551a4599bb7291ef9182d765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:31.267351 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cea676c551a4599bb7291ef9182d765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:31.267580 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cea676c551a4599bb7291ef9182d765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:31.270182 7377 tablet_bootstrap.cc:492] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: Bootstrap starting.
I20260809 01:52:31.270323 7374 tablet_bootstrap.cc:492] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: Bootstrap starting.
I20260809 01:52:31.271056 7377 tablet_bootstrap.cc:654] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:31.271199 7374 tablet_bootstrap.cc:654] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:31.271713 7376 tablet_bootstrap.cc:492] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: Bootstrap starting.
I20260809 01:52:31.272398 7377 tablet_bootstrap.cc:492] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: No bootstrap required, opened a new log
I20260809 01:52:31.272550 7377 ts_tablet_manager.cc:1397] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:31.272845 7376 tablet_bootstrap.cc:654] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:31.273135 7377 raft_consensus.cc:348] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.273598 7377 raft_consensus.cc:374] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:31.273665 7377 raft_consensus.cc:729] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc5a5bbf0d542f1b2341066f8ea36f6, State: Initialized, Role: FOLLOWER
I20260809 01:52:31.273844 7374 tablet_bootstrap.cc:492] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: No bootstrap required, opened a new log
I20260809 01:52:31.273842 7377 consensus_queue.cc:260] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.273960 7374 ts_tablet_manager.cc:1397] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 01:52:31.274257 7377 ts_tablet_manager.cc:1428] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:31.274643 7376 tablet_bootstrap.cc:492] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: No bootstrap required, opened a new log
I20260809 01:52:31.274641 7374 raft_consensus.cc:348] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.274782 7374 raft_consensus.cc:374] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:31.274816 7376 ts_tablet_manager.cc:1397] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:31.274827 7374 raft_consensus.cc:729] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167d92587818462b8f08b39a8f3806bc, State: Initialized, Role: FOLLOWER
I20260809 01:52:31.274959 7374 consensus_queue.cc:260] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.275396 7374 ts_tablet_manager.cc:1428] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:31.276103 7376 raft_consensus.cc:348] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.276261 7376 raft_consensus.cc:374] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:31.276307 7376 raft_consensus.cc:729] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d983f31c244226b419087158729032, State: Initialized, Role: FOLLOWER
I20260809 01:52:31.276504 7376 consensus_queue.cc:260] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.276887 7376 ts_tablet_manager.cc:1428] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260809 01:52:31.406512 6805 tablet.cc:2369] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:31.482015 7386 raft_consensus.cc:482] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:31.482177 7386 raft_consensus.cc:504] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.483239 7386 leader_election.cc:288] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233), 82d983f31c244226b419087158729032 (127.3.220.2:38169)
I20260809 01:52:31.487891 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.483648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:35185 (local address 127.3.220.1:45233)
0809 01:52:31.483983 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:31.483993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:31.484046 (+ 53us) server_negotiation.cc:407] Connection header received
0809 01:52:31.484526 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:31.484529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:31.484573 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:31.484681 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:31.484999 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.485653 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.486880 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.487289 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.487411 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.487481 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:31.487659 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 01:52:31.487742 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":110,"threads_started":1}
I20260809 01:52:31.487952 7388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.483563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:45233 (local address 127.3.220.3:35185)
0809 01:52:31.483956 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:31.483971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:31.484133 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:31.484705 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:31.484708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:31.484727 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:31.484877 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:31.484885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.485778 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.485782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:31.486749 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:31.486760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.487413 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.487416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:31.487515 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.487542 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:52:31.487639 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 01:52:31.487713 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":163,"threads_started":1}
I20260809 01:52:31.487982 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.484137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38169 (local address 127.3.220.3:55943)
0809 01:52:31.484482 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:31.484498 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:31.484590 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:31.484871 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:31.484873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:31.484890 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:31.485029 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:31.485034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.485884 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.485888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:31.486787 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:31.486796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.487452 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.487455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:31.487565 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.487594 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 01:52:31.487690 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 01:52:31.487763 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":4,"thread_start_us":130,"threads_started":1}
I20260809 01:52:31.488168 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.484306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:55943 (local address 127.3.220.2:38169)
0809 01:52:31.484598 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:31.484607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:31.484641 (+ 34us) server_negotiation.cc:407] Connection header received
0809 01:52:31.484705 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:31.484707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:31.484747 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:31.484843 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:31.485150 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.485753 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.486903 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.487326 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.487446 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.487509 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:31.487708 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 01:52:31.487792 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
I20260809 01:52:31.488622 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea676c551a4599bb7291ef9182d765" candidate_uuid: "167d92587818462b8f08b39a8f3806bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d983f31c244226b419087158729032" is_pre_election: true
I20260809 01:52:31.488823 6942 raft_consensus.cc:2454] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 167d92587818462b8f08b39a8f3806bc in term 0.
I20260809 01:52:31.488827 6736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea676c551a4599bb7291ef9182d765" candidate_uuid: "167d92587818462b8f08b39a8f3806bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" is_pre_election: true
I20260809 01:52:31.488992 6736 raft_consensus.cc:2454] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 167d92587818462b8f08b39a8f3806bc in term 0.
I20260809 01:52:31.489281 7037 leader_election.cc:302] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167d92587818462b8f08b39a8f3806bc, 82d983f31c244226b419087158729032; no voters:
I20260809 01:52:31.489625 7386 raft_consensus.cc:2781] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:31.489766 7386 raft_consensus.cc:482] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:31.489835 7386 raft_consensus.cc:3037] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:31.490949 7386 raft_consensus.cc:504] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.491300 7386 leader_election.cc:288] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [CANDIDATE]: Term 1 election: Requested vote from peers fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233), 82d983f31c244226b419087158729032 (127.3.220.2:38169)
I20260809 01:52:31.491667 6736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea676c551a4599bb7291ef9182d765" candidate_uuid: "167d92587818462b8f08b39a8f3806bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6"
I20260809 01:52:31.491813 6736 raft_consensus.cc:3037] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:31.491793 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cea676c551a4599bb7291ef9182d765" candidate_uuid: "167d92587818462b8f08b39a8f3806bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d983f31c244226b419087158729032"
I20260809 01:52:31.491928 6942 raft_consensus.cc:3037] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:31.492930 6736 raft_consensus.cc:2454] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 167d92587818462b8f08b39a8f3806bc in term 1.
I20260809 01:52:31.492961 6942 raft_consensus.cc:2454] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 167d92587818462b8f08b39a8f3806bc in term 1.
I20260809 01:52:31.493243 7037 leader_election.cc:302] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167d92587818462b8f08b39a8f3806bc, fbc5a5bbf0d542f1b2341066f8ea36f6; no voters:
I20260809 01:52:31.493436 7386 raft_consensus.cc:2781] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:31.493772 7386 raft_consensus.cc:686] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 1 LEADER]: Becoming Leader. State: Replica: 167d92587818462b8f08b39a8f3806bc, State: Running, Role: LEADER
I20260809 01:52:31.494035 7386 consensus_queue.cc:237] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.496384 6501 catalog_manager.cc:5495] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 167d92587818462b8f08b39a8f3806bc (127.3.220.3). New cstate: current_term: 1 leader_uuid: "167d92587818462b8f08b39a8f3806bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } health_report { overall_health: HEALTHY } } }
I20260809 01:52:31.514160 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.510753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.3.220.3:33567)
0809 01:52:31.511013 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:31.511020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:31.511032 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:52:31.511100 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:31.511102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:31.511151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:31.511261 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:31.511470 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.511994 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.512782 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.513019 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.513093 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.513318 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:31.513840 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:31.513930 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:31.513993 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 01:52:31.514039 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":123,"threads_started":1}
I20260809 01:52:31.523644 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.520749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:34755 (local address 127.3.220.1:45233)
0809 01:52:31.520892 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:31.520899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:31.520918 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:52:31.521011 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:31.521014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:31.521068 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:31.521157 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:31.521380 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.521951 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.522752 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.522955 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.523029 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.523285 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:31.523346 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:31.523399 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:31.523494 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 01:52:31.523540 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 01:52:31.523737 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.520506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:45233 (local address 127.3.220.3:34755)
0809 01:52:31.520818 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:31.520833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:31.520918 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:31.521177 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:31.521180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:31.521188 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:31.521288 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:31.521292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.522044 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.522046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:31.522643 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:31.522651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.523062 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.523064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:31.523134 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.523173 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:31.523417 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:31.523421 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:52:31.523470 (+ 49us) client_negotiation.cc:240] Negotiation successful
0809 01:52:31.523570 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
I20260809 01:52:31.529803 6736 raft_consensus.cc:1261] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Refusing update from remote peer 167d92587818462b8f08b39a8f3806bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:31.529798 6942 raft_consensus.cc:1261] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Refusing update from remote peer 167d92587818462b8f08b39a8f3806bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:31.530252 7386 consensus_queue.cc:1035] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:31.530459 7395 consensus_queue.cc:1035] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:31.548885 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.544712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:33567 (local address 127.3.220.1:34719)
0809 01:52:31.545089 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:31.545105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:31.545224 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:31.545672 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:31.545675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:31.545684 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:52:31.545790 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:31.545795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.546538 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.546541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:31.547525 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:31.547538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.548150 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.548154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:31.548243 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.548280 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:52:31.548593 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:52:31.548602 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 01:52:31.548667 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 01:52:31.548758 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
I20260809 01:52:31.548873 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.544766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:34719 (local address 127.3.220.3:33567)
0809 01:52:31.544886 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:31.544892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:31.545410 (+ 518us) server_negotiation.cc:407] Connection header received
0809 01:52:31.545468 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:31.545470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:31.545512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:31.545635 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:31.545876 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.546447 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.547637 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.548008 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.548140 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.548412 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:31.548493 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:31.548578 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:31.548677 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 01:52:31.548732 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:52:31.571228 6502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:31.571409 6502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:31.573546 6502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: Sending DeleteTablet for 3 replicas of tablet 4cea676c551a4599bb7291ef9182d765
I20260809 01:52:31.574368 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cea676c551a4599bb7291ef9182d765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:31 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:46126
I20260809 01:52:31.574383 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cea676c551a4599bb7291ef9182d765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:31 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:41006
I20260809 01:52:31.574563 7361 tablet_replica.cc:330] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: stopping tablet replica
I20260809 01:52:31.574705 7361 raft_consensus.cc:2229] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:31.574782 7359 tablet_replica.cc:330] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: stopping tablet replica
I20260809 01:52:31.574833 7361 raft_consensus.cc:2258] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:31.574903 7359 raft_consensus.cc:2229] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:31.575009 7359 raft_consensus.cc:2258] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:31.575178 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:52:31.575261 7111 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cea676c551a4599bb7291ef9182d765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:31 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:43322
I20260809 01:52:31.575429 7360 tablet_replica.cc:330] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: stopping tablet replica
I20260809 01:52:31.575528 7360 raft_consensus.cc:2229] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 1 LEADER]: Raft consensus shutting down.
W20260809 01:52:31.575598 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:31.575826 7360 raft_consensus.cc:2258] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:31.575850 7359 ts_tablet_manager.cc:1905] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:31.576844 7360 ts_tablet_manager.cc:1905] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:31.578696 7359 ts_tablet_manager.cc:1918] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:31.578800 7359 log.cc:1196] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/4cea676c551a4599bb7291ef9182d765
I20260809 01:52:31.579083 7359 ts_tablet_manager.cc:1939] T 4cea676c551a4599bb7291ef9182d765 P 82d983f31c244226b419087158729032: Deleting consensus metadata
I20260809 01:52:31.579478 7360 ts_tablet_manager.cc:1918] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:31.579546 7360 log.cc:1196] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/4cea676c551a4599bb7291ef9182d765
I20260809 01:52:31.579797 7360 ts_tablet_manager.cc:1939] T 4cea676c551a4599bb7291ef9182d765 P 167d92587818462b8f08b39a8f3806bc: Deleting consensus metadata
I20260809 01:52:31.579754 6484 catalog_manager.cc:4841] TS 82d983f31c244226b419087158729032 (127.3.220.2:38169): tablet 4cea676c551a4599bb7291ef9182d765 (table test-table [id=e6716c49f8ed4d52a6f1fd5bd630457b]) successfully deleted
I20260809 01:52:31.580370 6483 catalog_manager.cc:4841] TS 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567): tablet 4cea676c551a4599bb7291ef9182d765 (table test-table [id=e6716c49f8ed4d52a6f1fd5bd630457b]) successfully deleted
I20260809 01:52:31.580659 7111 tablet_service.cc:1467] Processing CreateTablet for tablet d7250d0c55a240189716d1bfe919d22c (DEFAULT_TABLE table=test-table [id=9f0ad195358243e6ba5529a6cb4c2bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:31.580663 6915 tablet_service.cc:1467] Processing CreateTablet for tablet d7250d0c55a240189716d1bfe919d22c (DEFAULT_TABLE table=test-table [id=9f0ad195358243e6ba5529a6cb4c2bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:31.580897 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7250d0c55a240189716d1bfe919d22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:31.580910 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7250d0c55a240189716d1bfe919d22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:31.582253 7361 ts_tablet_manager.cc:1905] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:31.583029 7374 tablet_bootstrap.cc:492] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: Bootstrap starting.
I20260809 01:52:31.583146 7376 tablet_bootstrap.cc:492] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: Bootstrap starting.
I20260809 01:52:31.583937 7374 tablet_bootstrap.cc:654] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:31.584059 7376 tablet_bootstrap.cc:654] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:31.584937 6707 tablet_service.cc:1467] Processing CreateTablet for tablet d7250d0c55a240189716d1bfe919d22c (DEFAULT_TABLE table=test-table [id=9f0ad195358243e6ba5529a6cb4c2bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:31.585206 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7250d0c55a240189716d1bfe919d22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:31.585426 7376 tablet_bootstrap.cc:492] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: No bootstrap required, opened a new log
I20260809 01:52:31.585551 7376 ts_tablet_manager.cc:1397] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:31.585830 7374 tablet_bootstrap.cc:492] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: No bootstrap required, opened a new log
I20260809 01:52:31.585906 7374 ts_tablet_manager.cc:1397] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:31.585948 7376 raft_consensus.cc:348] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.586053 7376 raft_consensus.cc:374] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:31.586081 7376 raft_consensus.cc:729] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d983f31c244226b419087158729032, State: Initialized, Role: FOLLOWER
I20260809 01:52:31.586228 7376 consensus_queue.cc:260] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.586313 7374 raft_consensus.cc:348] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.586409 7374 raft_consensus.cc:374] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:31.586439 7374 raft_consensus.cc:729] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167d92587818462b8f08b39a8f3806bc, State: Initialized, Role: FOLLOWER
I20260809 01:52:31.586552 7376 ts_tablet_manager.cc:1428] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:31.586535 7374 consensus_queue.cc:260] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.586779 7374 ts_tablet_manager.cc:1428] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:31.587442 7377 tablet_bootstrap.cc:492] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6: Bootstrap starting.
I20260809 01:52:31.588361 7377 tablet_bootstrap.cc:654] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:31.589026 7361 ts_tablet_manager.cc:1918] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:52:31.589102 7361 log.cc:1196] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/4cea676c551a4599bb7291ef9182d765
I20260809 01:52:31.589480 7361 ts_tablet_manager.cc:1939] T 4cea676c551a4599bb7291ef9182d765 P fbc5a5bbf0d542f1b2341066f8ea36f6: Deleting consensus metadata
I20260809 01:52:31.589787 7377 tablet_bootstrap.cc:492] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6: No bootstrap required, opened a new log
I20260809 01:52:31.589895 7377 ts_tablet_manager.cc:1397] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:31.590268 6484 catalog_manager.cc:4841] TS fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233): tablet 4cea676c551a4599bb7291ef9182d765 (table test-table [id=e6716c49f8ed4d52a6f1fd5bd630457b]) successfully deleted
I20260809 01:52:31.590324 7377 raft_consensus.cc:348] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.590442 7377 raft_consensus.cc:374] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:31.590482 7377 raft_consensus.cc:729] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc5a5bbf0d542f1b2341066f8ea36f6, State: Initialized, Role: FOLLOWER
I20260809 01:52:31.590606 7377 consensus_queue.cc:260] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.590884 7377 ts_tablet_manager.cc:1428] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 01:52:31.833772 7339 raft_consensus.cc:482] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:31.833956 7339 raft_consensus.cc:504] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.834478 7339 leader_election.cc:288] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d983f31c244226b419087158729032 (127.3.220.2:38169), 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567)
I20260809 01:52:31.834973 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7250d0c55a240189716d1bfe919d22c" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167d92587818462b8f08b39a8f3806bc" is_pre_election: true
I20260809 01:52:31.834992 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7250d0c55a240189716d1bfe919d22c" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d983f31c244226b419087158729032" is_pre_election: true
I20260809 01:52:31.835134 6942 raft_consensus.cc:2454] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 0.
I20260809 01:52:31.835135 7139 raft_consensus.cc:2454] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 0.
I20260809 01:52:31.835487 6643 leader_election.cc:302] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d983f31c244226b419087158729032, fbc5a5bbf0d542f1b2341066f8ea36f6; no voters:
I20260809 01:52:31.835678 7339 raft_consensus.cc:2781] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:31.835744 7339 raft_consensus.cc:482] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:31.835803 7339 raft_consensus.cc:3037] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:31.836858 7339 raft_consensus.cc:504] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.837245 7339 leader_election.cc:288] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 election: Requested vote from peers 82d983f31c244226b419087158729032 (127.3.220.2:38169), 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567)
I20260809 01:52:31.837618 6942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7250d0c55a240189716d1bfe919d22c" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d983f31c244226b419087158729032"
I20260809 01:52:31.837764 6942 raft_consensus.cc:3037] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:31.837777 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7250d0c55a240189716d1bfe919d22c" candidate_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167d92587818462b8f08b39a8f3806bc"
I20260809 01:52:31.837937 7139 raft_consensus.cc:3037] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:31.838759 6942 raft_consensus.cc:2454] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 1.
I20260809 01:52:31.838935 7139 raft_consensus.cc:2454] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc5a5bbf0d542f1b2341066f8ea36f6 in term 1.
I20260809 01:52:31.839071 6643 leader_election.cc:302] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d983f31c244226b419087158729032, fbc5a5bbf0d542f1b2341066f8ea36f6; no voters:
I20260809 01:52:31.839248 7339 raft_consensus.cc:2781] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:31.839345 7339 raft_consensus.cc:686] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [term 1 LEADER]: Becoming Leader. State: Replica: fbc5a5bbf0d542f1b2341066f8ea36f6, State: Running, Role: LEADER
I20260809 01:52:31.839493 7339 consensus_queue.cc:237] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } }
I20260809 01:52:31.841154 6500 catalog_manager.cc:5495] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5a5bbf0d542f1b2341066f8ea36f6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 45233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 } health_report { overall_health: UNKNOWN } } }
I20260809 01:52:31.894012 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:31.890639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.3.220.1:45233)
0809 01:52:31.890757 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:31.890761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:31.890827 (+ 66us) server_negotiation.cc:407] Connection header received
0809 01:52:31.890982 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:31.890988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:31.891037 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:31.891127 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:31.891379 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.891906 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.892836 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:31.893078 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:31.893190 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:31.893451 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:31.893584 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:31.893661 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:31.893795 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 01:52:31.893854 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 01:52:31.901007 6942 raft_consensus.cc:1261] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Refusing update from remote peer fbc5a5bbf0d542f1b2341066f8ea36f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:31.900995 7139 raft_consensus.cc:1261] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Refusing update from remote peer fbc5a5bbf0d542f1b2341066f8ea36f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:31.901362 7297 consensus_queue.cc:1035] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "167d92587818462b8f08b39a8f3806bc" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:31.901510 7339 consensus_queue.cc:1035] T d7250d0c55a240189716d1bfe919d22c P fbc5a5bbf0d542f1b2341066f8ea36f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d983f31c244226b419087158729032" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 01:52:31.915432 7328 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 01:52:33.467681 6500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:33.467854 6500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:33.470193 6500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5687742b4bb745a6b1d1bbe48ecfee31: Sending DeleteTablet for 3 replicas of tablet d7250d0c55a240189716d1bfe919d22c
I20260809 01:52:33.470679 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet d7250d0c55a240189716d1bfe919d22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:33 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:46126
I20260809 01:52:33.470813 7111 tablet_service.cc:1514] Processing DeleteTablet for tablet d7250d0c55a240189716d1bfe919d22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:33 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:43322
I20260809 01:52:33.471190 7451 tablet_replica.cc:330] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: stopping tablet replica
I20260809 01:52:33.471186 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 6615
I20260809 01:52:33.471180 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet d7250d0c55a240189716d1bfe919d22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:33 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:41006
I20260809 01:52:33.482817 7451 raft_consensus.cc:2229] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:33.483065 7451 raft_consensus.cc:2258] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:33.484798 7451 ts_tablet_manager.cc:1905] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:33.488631 7451 ts_tablet_manager.cc:1918] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 01:52:33.488719 7451 log.cc:1196] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/d7250d0c55a240189716d1bfe919d22c
I20260809 01:52:33.488986 7451 ts_tablet_manager.cc:1939] T d7250d0c55a240189716d1bfe919d22c P 167d92587818462b8f08b39a8f3806bc: Deleting consensus metadata
I20260809 01:52:33.489912 6483 catalog_manager.cc:4841] TS 167d92587818462b8f08b39a8f3806bc (127.3.220.3:33567): tablet d7250d0c55a240189716d1bfe919d22c (table test-table [id=9f0ad195358243e6ba5529a6cb4c2bc4]) successfully deleted
I20260809 01:52:33.491684 7452 tablet_replica.cc:330] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: stopping tablet replica
I20260809 01:52:33.491843 7452 raft_consensus.cc:2229] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:33.491998 7452 raft_consensus.cc:2258] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:33.493046 7452 ts_tablet_manager.cc:1905] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:33.496645 7452 ts_tablet_manager.cc:1918] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 01:52:33.496758 7452 log.cc:1196] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/d7250d0c55a240189716d1bfe919d22c
I20260809 01:52:33.497040 7452 ts_tablet_manager.cc:1939] T d7250d0c55a240189716d1bfe919d22c P 82d983f31c244226b419087158729032: Deleting consensus metadata
I20260809 01:52:33.497929 6484 catalog_manager.cc:4841] TS 82d983f31c244226b419087158729032 (127.3.220.2:38169): tablet d7250d0c55a240189716d1bfe919d22c (table test-table [id=9f0ad195358243e6ba5529a6cb4c2bc4]) successfully deleted
W20260809 01:52:33.502731 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:45233 (error 108)
I20260809 01:52:33.503119 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 6820
I20260809 01:52:33.511394 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:33.509476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:45233 (local address 127.0.0.1:46160)
0809 01:52:33.511221 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:33.511298 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:45233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1665,"thread_start_us":346,"threads_started":1}
W20260809 01:52:33.511545 6484 catalog_manager.cc:4584] TS fbc5a5bbf0d542f1b2341066f8ea36f6 (127.3.220.1:45233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7250d0c55a240189716d1bfe919d22c: Network error: Client connection negotiation failed: client connection to 127.3.220.1:45233: connect: Connection refused (error 111)
I20260809 01:52:33.518608 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 7013
I20260809 01:52:33.532832 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 6457
2026-08-09T01:52:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6975 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:33 dist-test-slave-gkw7 krb5kdc[7458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:33 dist-test-slave-gkw7 krb5kdc[7458](info): set up 2 sockets
Aug 09 01:52:33 dist-test-slave-gkw7 krb5kdc[7458](info): commencing operation
krb5kdc: starting...
W20260809 01:52:34.768235 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:34 dist-test-slave-gkw7 krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:34.898784 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:45147
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:45147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:35.011457 7476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:35.011777 7476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:35.011852 7476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:35.015872 7476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:35.015957 7476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:35.015996 7476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:35.016055 7476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:35.016096 7476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:35.021322 7476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:45147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:45147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:35.022481 7476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:35.023538 7476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:35.032446 7491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:35.032464 7487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:35.032483 7488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:35.033046 7476 server_base.cc:1034] running on GCE node
I20260809 01:52:35.033691 7476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:35.034803 7476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:35.036000 7476 hybrid_clock.cc:648] HybridClock initialized: now 1786240355035986 us; error 28 us; skew 500 ppm
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:35.038837 7476 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:35.039788 7476 webserver.cc:460] Webserver started at http://127.3.220.62:34523/ using document root <none> and password file <none>
I20260809 01:52:35.040369 7476 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:35.040465 7476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:35.040705 7476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:35.042869 7476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "83c7d187077c421284995a732cf9451a"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "d3a94e42574ae442dc277bf8922c8a4e"
server_key_iv: "0b7ab2833dab34e1f919c1ba6a591b2f"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.043455 7476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "83c7d187077c421284995a732cf9451a"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "d3a94e42574ae442dc277bf8922c8a4e"
server_key_iv: "0b7ab2833dab34e1f919c1ba6a591b2f"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.047215 7476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:52:35.049666 7500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.050947 7476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:35.051095 7476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "83c7d187077c421284995a732cf9451a"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "d3a94e42574ae442dc277bf8922c8a4e"
server_key_iv: "0b7ab2833dab34e1f919c1ba6a591b2f"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.051213 7476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:35.073364 7476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:35.076489 7476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:35.076699 7476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:35.084050 7476 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:45147
I20260809 01:52:35.084079 7576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:45147 every 8 connection(s)
I20260809 01:52:35.085014 7476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:35.085927 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 7476
I20260809 01:52:35.086046 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:35.086376 3952 external_mini_cluster.cc:1428] Setting key f98364687d60ce68f60d51d2b806a064
I20260809 01:52:35.088464 7579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:35.094269 7579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a: Bootstrap starting.
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[7458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:35.096930 7579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:35.097812 7579 log.cc:824] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:35.099733 7579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a: No bootstrap required, opened a new log
I20260809 01:52:35.100829 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:35.087872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.3.220.62:45147)
0809 01:52:35.088326 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:35.088342 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:35.088379 (+ 37us) server_negotiation.cc:407] Connection header received
0809 01:52:35.089091 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:35.089107 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:35.089439 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:35.089850 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:35.090718 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.091718 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.092599 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.093083 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.093261 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:35.096190 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:35.096195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:35.096208 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:35.096237 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:35.098111 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:35.098566 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:35.098571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:35.098576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:35.098692 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:35.098977 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:35.098981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:35.098982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:35.099382 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:35.099595 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:35.099836 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 01:52:35.100132 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":160,"threads_started":1}
I20260809 01:52:35.103089 7579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83c7d187077c421284995a732cf9451a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45147 } }
I20260809 01:52:35.103363 7579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:35.103454 7579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c7d187077c421284995a732cf9451a, State: Initialized, Role: FOLLOWER
I20260809 01:52:35.103904 7579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83c7d187077c421284995a732cf9451a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45147 } }
I20260809 01:52:35.104028 7579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:35.104112 7579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:35.104238 7579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:35.105135 7579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83c7d187077c421284995a732cf9451a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45147 } }
I20260809 01:52:35.105480 7579 leader_election.cc:302] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83c7d187077c421284995a732cf9451a; no voters:
I20260809 01:52:35.105866 7579 leader_election.cc:288] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:35.106007 7585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:35.106262 7585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [term 1 LEADER]: Becoming Leader. State: Replica: 83c7d187077c421284995a732cf9451a, State: Running, Role: LEADER
I20260809 01:52:35.106616 7585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83c7d187077c421284995a732cf9451a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45147 } }
I20260809 01:52:35.107090 7579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:35.108350 7587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83c7d187077c421284995a732cf9451a. Latest consensus state: current_term: 1 leader_uuid: "83c7d187077c421284995a732cf9451a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83c7d187077c421284995a732cf9451a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45147 } } }
I20260809 01:52:35.108486 7587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:35.108816 7586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83c7d187077c421284995a732cf9451a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83c7d187077c421284995a732cf9451a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45147 } } }
I20260809 01:52:35.108922 7586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:35.109081 7597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:35.111788 7597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:35.117340 7597 catalog_manager.cc:1347] Generated new cluster ID: 6c13df0b17114aca9bd8f230a1445681
I20260809 01:52:35.117470 7597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:35.125360 7597 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:35.126583 7597 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:35.132777 7597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83c7d187077c421284995a732cf9451a: Generated new TSK 0
I20260809 01:52:35.133590 7597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:35.164992 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45147
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:35.284525 7612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:35.284822 7612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:35.284906 7612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:35.289073 7612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:35.289165 7612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:35.289258 7612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:35.294654 7612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:35.295765 7612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:35.296829 7612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:35.306788 7622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:35.306869 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:35.306898 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:35.307183 7612 server_base.cc:1034] running on GCE node
I20260809 01:52:35.307619 7612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:35.308408 7612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:35.309646 7612 hybrid_clock.cc:648] HybridClock initialized: now 1786240355309604 us; error 46 us; skew 500 ppm
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:35.312808 7612 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:35.313799 7612 webserver.cc:460] Webserver started at http://127.3.220.1:46345/ using document root <none> and password file <none>
I20260809 01:52:35.314335 7612 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:35.314401 7612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:35.314574 7612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:35.316582 7612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "6d6eedc814864d679f9cccd5470460ab"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "5373bce673b26c9948c0d9f25d07d4d6"
server_key_iv: "bc2ea28ee97f44f74f9382cae4e36c95"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.317147 7612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "6d6eedc814864d679f9cccd5470460ab"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "5373bce673b26c9948c0d9f25d07d4d6"
server_key_iv: "bc2ea28ee97f44f74f9382cae4e36c95"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.320976 7612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 01:52:35.323601 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.325006 7612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:35.325105 7612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "6d6eedc814864d679f9cccd5470460ab"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "5373bce673b26c9948c0d9f25d07d4d6"
server_key_iv: "bc2ea28ee97f44f74f9382cae4e36c95"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.325201 7612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:35.337842 7612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:35.341133 7612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:35.341291 7612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:35.341924 7612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:35.342713 7612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:35.342760 7612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.342799 7612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:35.342815 7612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.351999 7612 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:39735
I20260809 01:52:35.352022 7798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:39735 every 8 connection(s)
I20260809 01:52:35.352888 7612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:35.362677 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 7612
I20260809 01:52:35.362807 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[7458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:35.363185 3952 external_mini_cluster.cc:1428] Setting key 795996cc599846b362eaf3d8772dfefc
I20260809 01:52:35.367493 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:35.354738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:47563 (local address 127.3.220.62:45147)
0809 01:52:35.354880 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:35.354884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:35.355913 (+ 1029us) server_negotiation.cc:407] Connection header received
0809 01:52:35.357282 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:35.357285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:35.357344 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:35.357454 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:35.358635 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.359183 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.360343 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.360669 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.360834 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:35.363710 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:35.363714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:35.363717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:35.363751 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:35.365283 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:35.365917 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:35.365922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:35.365924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:35.366011 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:35.366375 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:35.366378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:35.366380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:35.366564 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:35.366683 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:35.367194 (+ 511us) server_negotiation.cc:299] Negotiation successful
0809 01:52:35.367323 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 01:52:35.368361 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:35.355105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45147 (local address 127.3.220.1:47563)
0809 01:52:35.355755 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:35.355812 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:35.357052 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:35.357660 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:35.357674 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:35.358022 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:35.358481 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:35.358495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.359316 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.359319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:35.360200 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:35.360209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.360878 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.360883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:35.361078 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:35.361771 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:35.361788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:35.363531 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:35.365412 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:35.365418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:35.365434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:35.365803 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:35.366107 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:35.366111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:35.366113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:35.366267 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:35.366690 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:35.366698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:35.366942 (+ 244us) client_negotiation.cc:769] Sending connection context
0809 01:52:35.367127 (+ 185us) client_negotiation.cc:240] Negotiation successful
0809 01:52:35.367446 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260809 01:52:35.369720 7799 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45147
I20260809 01:52:35.369976 7799 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:35.370455 7799 heartbeater.cc:511] Master 127.3.220.62:45147 requested a full tablet report, sending...
I20260809 01:52:35.372205 7526 ts_manager.cc:194] Registered new tserver with Master: 6d6eedc814864d679f9cccd5470460ab (127.3.220.1:39735)
I20260809 01:52:35.373912 7526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:47563
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:35.396489 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45147
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:35.511466 7805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:35.511730 7805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:35.511773 7805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:35.516353 7805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:35.516419 7805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:35.516515 7805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:35.521862 7805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:35.522977 7805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:35.524086 7805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:35.534242 7813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:35.534294 7805 server_base.cc:1034] running on GCE node
W20260809 01:52:35.534271 7814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:35.534401 7816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:35.534827 7805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:35.535491 7805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:35.536661 7805 hybrid_clock.cc:648] HybridClock initialized: now 1786240355536647 us; error 26 us; skew 500 ppm
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:35.539582 7805 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:35.540524 7805 webserver.cc:460] Webserver started at http://127.3.220.2:43437/ using document root <none> and password file <none>
I20260809 01:52:35.541093 7805 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:35.541153 7805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:35.541378 7805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:35.543568 7805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "e266d516bd6c478cbd0f40954b4d464b"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "84559d1afede3308b27a5fcfd0ce08bf"
server_key_iv: "35f0e4ecade7a2137f5407df12e77397"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.544189 7805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "e266d516bd6c478cbd0f40954b4d464b"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "84559d1afede3308b27a5fcfd0ce08bf"
server_key_iv: "35f0e4ecade7a2137f5407df12e77397"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.548116 7805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 01:52:35.550800 7829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.552080 7805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:35.552242 7805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "e266d516bd6c478cbd0f40954b4d464b"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "84559d1afede3308b27a5fcfd0ce08bf"
server_key_iv: "35f0e4ecade7a2137f5407df12e77397"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.552371 7805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:35.566712 7805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:35.569686 7805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:35.569895 7805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:35.570492 7805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:35.571327 7805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:35.571375 7805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.571439 7805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:35.571475 7805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.581082 7805 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:40511
I20260809 01:52:35.581142 7991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:40511 every 8 connection(s)
I20260809 01:52:35.582142 7805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:35.584208 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 7805
I20260809 01:52:35.584334 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:35.584687 3952 external_mini_cluster.cc:1428] Setting key ae7fb730d4f41922985075e5fae42295
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[7458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:52:35.597462 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:35.584029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:47869 (local address 127.3.220.62:45147)
0809 01:52:35.584161 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:35.584170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:35.585306 (+ 1136us) server_negotiation.cc:407] Connection header received
0809 01:52:35.586996 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:35.587002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:35.587073 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:35.587208 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:35.588406 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.589291 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.590460 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.590701 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.590814 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:35.593439 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:35.593443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:35.593446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:35.593478 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:35.595090 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:35.595678 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:35.595682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:35.595684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:35.595777 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:35.596174 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:35.596178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:35.596180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:35.596454 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:35.596572 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:35.597131 (+ 559us) server_negotiation.cc:299] Negotiation successful
0809 01:52:35.597267 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:52:35.598515 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:35.584575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45147 (local address 127.3.220.2:47869)
0809 01:52:35.585075 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:35.585153 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:35.586758 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:35.587365 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:35.587371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:35.587721 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:35.588194 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:35.588208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.589416 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.589418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:35.590316 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:35.590325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.590851 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.590855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:35.591020 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:35.591705 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:35.591722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:35.593263 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:35.595218 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:35.595224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:35.595238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:35.595564 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:35.595885 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:35.595888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:35.595890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:35.596050 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:35.596570 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:35.596577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:35.596864 (+ 287us) client_negotiation.cc:769] Sending connection context
0809 01:52:35.597065 (+ 201us) client_negotiation.cc:240] Negotiation successful
0809 01:52:35.597406 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":146,"threads_started":1}
I20260809 01:52:35.599822 7992 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45147
I20260809 01:52:35.600073 7992 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:35.600689 7992 heartbeater.cc:511] Master 127.3.220.62:45147 requested a full tablet report, sending...
I20260809 01:52:35.601987 7526 ts_manager.cc:194] Registered new tserver with Master: e266d516bd6c478cbd0f40954b4d464b (127.3.220.2:40511)
I20260809 01:52:35.602602 7526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:47869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:35.614882 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45147
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:35.731395 8001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:35.731642 8001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:35.731685 8001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:35.735651 8001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:35.735705 8001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:35.735761 8001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:35.740954 8001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:35.742020 8001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:35.743013 8001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:35.752871 8011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:35.752919 8013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:35.752857 8010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:35.753059 8001 server_base.cc:1034] running on GCE node
I20260809 01:52:35.753595 8001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:35.754272 8001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:35.755487 8001 hybrid_clock.cc:648] HybridClock initialized: now 1786240355755453 us; error 47 us; skew 500 ppm
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:35.758616 8001 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:35.759506 8001 webserver.cc:460] Webserver started at http://127.3.220.3:33561/ using document root <none> and password file <none>
I20260809 01:52:35.760073 8001 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:35.760134 8001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:35.760365 8001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:35.762495 8001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "4f93d5e05b40476b972a09de2ebe93af"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "759eb6a546ac4a89452e163fd665c6b8"
server_key_iv: "fa48eecc0c3d109da33edf5a21316311"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.763077 8001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "4f93d5e05b40476b972a09de2ebe93af"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "759eb6a546ac4a89452e163fd665c6b8"
server_key_iv: "fa48eecc0c3d109da33edf5a21316311"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.766904 8001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:52:35.769336 8021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.770597 8001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:35.770735 8001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "4f93d5e05b40476b972a09de2ebe93af"
format_stamp: "Formatted at 2026-08-09 01:52:35 on dist-test-slave-gkw7"
server_key: "759eb6a546ac4a89452e163fd665c6b8"
server_key_iv: "fa48eecc0c3d109da33edf5a21316311"
server_key_version: "encryptionkey@0"
I20260809 01:52:35.770848 8001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:35.806885 8001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:35.810460 8001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:35.810681 8001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:35.811342 8001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:35.812199 8001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:35.812271 8001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.812346 8001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:35.812399 8001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:35.821286 8001 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:39335
I20260809 01:52:35.821317 8181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:39335 every 8 connection(s)
I20260809 01:52:35.822285 8001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:35.832068 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 8001
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[7458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:35.832218 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:35.832576 3952 external_mini_cluster.cc:1428] Setting key 5fb49c8f6c8660a36f043c15fc4fec92
I20260809 01:52:35.836700 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:35.824224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:59917 (local address 127.3.220.62:45147)
0809 01:52:35.824367 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:35.824371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:35.825202 (+ 831us) server_negotiation.cc:407] Connection header received
0809 01:52:35.826646 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:35.826649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:35.826706 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:35.826814 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:35.827961 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.828615 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.829963 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.830192 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.830328 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:35.832873 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:35.832876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:35.832879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:35.832925 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:35.834723 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:35.835338 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:35.835341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:35.835342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:35.835421 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:35.835733 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:35.835735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:35.835736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:35.835905 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:35.835978 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:35.836458 (+ 480us) server_negotiation.cc:299] Negotiation successful
0809 01:52:35.836563 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 01:52:35.837464 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:35.824568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45147 (local address 127.3.220.3:59917)
0809 01:52:35.825046 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:35.825089 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:35.826423 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:35.826985 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:35.826992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:35.827344 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:35.827818 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:35.827830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.828745 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.828755 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:35.829839 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:35.829849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:35.830372 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:35.830375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:35.830542 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:35.831085 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:35.831100 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:35.832705 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:35.834870 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:35.834879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:35.834897 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:35.835245 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:35.835516 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:35.835518 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:35.835519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:35.835647 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:35.835991 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:35.835995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:35.836216 (+ 221us) client_negotiation.cc:769] Sending connection context
0809 01:52:35.836372 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 01:52:35.836615 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260809 01:52:35.838665 8182 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45147
I20260809 01:52:35.838925 8182 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:35.839511 8182 heartbeater.cc:511] Master 127.3.220.62:45147 requested a full tablet report, sending...
I20260809 01:52:35.840746 7526 ts_manager.cc:194] Registered new tserver with Master: 4f93d5e05b40476b972a09de2ebe93af (127.3.220.3:39335)
I20260809 01:52:35.841343 7526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:59917
I20260809 01:52:35.846814 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 01:52:35.862424 7582 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:35.860705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.3.220.62:45147)
0809 01:52:35.860894 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:35.860899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:35.861060 (+ 161us) server_negotiation.cc:407] Connection header received
0809 01:52:35.861278 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:35.861281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:35.861327 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:35.861418 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:35.862252 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60872: BlockingRecv error: recv got EOF from 127.0.0.1:60872 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 01:52:35.863423 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 7612
I20260809 01:52:35.880515 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 7805
I20260809 01:52:35.901098 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 8001
I20260809 01:52:35.917879 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 7476
2026-08-09T01:52:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2379 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[8198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[8198](info): set up 2 sockets
Aug 09 01:52:35 dist-test-slave-gkw7 krb5kdc[8198](info): commencing operation
krb5kdc: starting...
W20260809 01:52:37.159652 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:37 dist-test-slave-gkw7 krb5kdc[8198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:37.292725 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:36463
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:36463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:37.405676 8221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:37.405959 8221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:37.406000 8221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:37.410036 8221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:37.410096 8221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:37.410116 8221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:37.410133 8221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:37.410154 8221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:37.415478 8221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:36463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:36463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:37.416602 8221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:37.417671 8221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:37.426278 8230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:37.426244 8226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:37.426527 8221 server_base.cc:1034] running on GCE node
W20260809 01:52:37.426293 8227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:37.427201 8221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:37.428298 8221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:37.429486 8221 hybrid_clock.cc:648] HybridClock initialized: now 1786240357429455 us; error 44 us; skew 500 ppm
Aug 09 01:52:37 dist-test-slave-gkw7 krb5kdc[8198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240357, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:37.432251 8221 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:37.433120 8221 webserver.cc:460] Webserver started at http://127.3.220.62:36251/ using document root <none> and password file <none>
I20260809 01:52:37.433774 8221 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:37.433869 8221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:37.434109 8221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:37.436137 8221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "9abd97f4a9bc4095951281fe117a80d1"
format_stamp: "Formatted at 2026-08-09 01:52:37 on dist-test-slave-gkw7"
server_key: "0ae328128703bfed224b610bc1705dd3"
server_key_iv: "58c17987452e048045b4abbd97028091"
server_key_version: "encryptionkey@0"
I20260809 01:52:37.436722 8221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "9abd97f4a9bc4095951281fe117a80d1"
format_stamp: "Formatted at 2026-08-09 01:52:37 on dist-test-slave-gkw7"
server_key: "0ae328128703bfed224b610bc1705dd3"
server_key_iv: "58c17987452e048045b4abbd97028091"
server_key_version: "encryptionkey@0"
I20260809 01:52:37.440550 8221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 01:52:37.443003 8239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:37.444207 8221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 01:52:37.444356 8221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "9abd97f4a9bc4095951281fe117a80d1"
format_stamp: "Formatted at 2026-08-09 01:52:37 on dist-test-slave-gkw7"
server_key: "0ae328128703bfed224b610bc1705dd3"
server_key_iv: "58c17987452e048045b4abbd97028091"
server_key_version: "encryptionkey@0"
I20260809 01:52:37.444466 8221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:37.491783 8221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:37.495227 8221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:37.495458 8221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:37.503219 8221 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:36463
I20260809 01:52:37.503598 8310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:36463 every 8 connection(s)
I20260809 01:52:37.504248 8221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:37.507212 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:37.510232 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 8221
I20260809 01:52:37.510342 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:37.510610 3952 external_mini_cluster.cc:1428] Setting key 20c90238ad2995c708614b21eb5a77f9
I20260809 01:52:37.513612 8312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1: Bootstrap starting.
I20260809 01:52:37.516352 8312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:37.517112 8312 log.cc:824] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:37.519264 8312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1: No bootstrap required, opened a new log
Aug 09 01:52:37 dist-test-slave-gkw7 krb5kdc[8198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:37.522842 8312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abd97f4a9bc4095951281fe117a80d1" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36463 } }
I20260809 01:52:37.523085 8312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:37.523168 8312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abd97f4a9bc4095951281fe117a80d1, State: Initialized, Role: FOLLOWER
I20260809 01:52:37.523624 8312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abd97f4a9bc4095951281fe117a80d1" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36463 } }
I20260809 01:52:37.523782 8312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:37.523872 8312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:37.524008 8312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:37.524920 8312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abd97f4a9bc4095951281fe117a80d1" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36463 } }
I20260809 01:52:37.525279 8312 leader_election.cc:302] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abd97f4a9bc4095951281fe117a80d1; no voters:
I20260809 01:52:37.525677 8312 leader_election.cc:288] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:37.525763 8318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:37.526039 8318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [term 1 LEADER]: Becoming Leader. State: Replica: 9abd97f4a9bc4095951281fe117a80d1, State: Running, Role: LEADER
I20260809 01:52:37.526403 8318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abd97f4a9bc4095951281fe117a80d1" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36463 } }
I20260809 01:52:37.526798 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:37.512018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.3.220.62:36463)
0809 01:52:37.512479 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:37.512500 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:37.512550 (+ 50us) server_negotiation.cc:407] Connection header received
0809 01:52:37.513358 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:37.513381 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:37.513846 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:37.514219 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:37.514927 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:37.516384 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:37.517277 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:37.517793 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:37.517976 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:37.520854 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:37.520859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:37.520872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:37.520900 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:37.523538 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:37.524119 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:37.524125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:37.524132 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:37.524267 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:37.524567 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:37.524570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:37.524572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:37.524975 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:37.525180 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:37.525490 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 01:52:37.525866 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
I20260809 01:52:37.527103 8312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:37.528357 8319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abd97f4a9bc4095951281fe117a80d1. Latest consensus state: current_term: 1 leader_uuid: "9abd97f4a9bc4095951281fe117a80d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abd97f4a9bc4095951281fe117a80d1" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36463 } } }
I20260809 01:52:37.528407 8320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9abd97f4a9bc4095951281fe117a80d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abd97f4a9bc4095951281fe117a80d1" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36463 } } }
I20260809 01:52:37.528523 8320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:37.528510 8319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:37.529004 8322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:37.530280 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:52:37.535094 8322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:37.541682 8322 catalog_manager.cc:1347] Generated new cluster ID: 012bafbccbfc4bd2acb51a17399b6f11
I20260809 01:52:37.541751 8322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:37.555321 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:37.549449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.3.220.62:36463)
0809 01:52:37.549587 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:37.549591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:37.549912 (+ 321us) server_negotiation.cc:407] Connection header received
0809 01:52:37.550029 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:37.550033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:37.550088 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:37.550179 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:37.550772 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:37.551599 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:37.552773 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:37.553146 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:37.553254 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:37.554215 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:37.554219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:37.554222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:37.554833 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:37.554870 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:37.554891 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:37.554984 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:37.555095 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 01:52:37.555163 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 01:52:37.585286 8322 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:37.586419 8322 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:37.596143 8322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9abd97f4a9bc4095951281fe117a80d1: Generated new TSK 0
I20260809 01:52:37.596863 8322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:52:37.605775 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 8221
2026-08-09T01:52:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1690 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:37 dist-test-slave-gkw7 krb5kdc[8356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:37 dist-test-slave-gkw7 krb5kdc[8356](info): set up 2 sockets
Aug 09 01:52:37 dist-test-slave-gkw7 krb5kdc[8356](info): commencing operation
krb5kdc: starting...
W20260809 01:52:38.845939 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:38 dist-test-slave-gkw7 krb5kdc[8356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:38.982782 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41249
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:41249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:39.095566 8374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:39.095811 8374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:39.095851 8374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:39.099901 8374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:39.099954 8374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:39.099974 8374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:39.099992 8374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:39.100035 8374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:39.105230 8374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:41249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:39.106338 8374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:39.107365 8374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:39.116364 8381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:39.116503 8374 server_base.cc:1034] running on GCE node
W20260809 01:52:39.116374 8384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:39.116717 8380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:39.117350 8374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:39.118487 8374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:39.119668 8374 hybrid_clock.cc:648] HybridClock initialized: now 1786240359119660 us; error 30 us; skew 500 ppm
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:39.127744 8374 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:39.128667 8374 webserver.cc:460] Webserver started at http://127.3.220.62:45857/ using document root <none> and password file <none>
I20260809 01:52:39.129251 8374 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:39.129314 8374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:39.129629 8374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:39.131726 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "16e0fe7138d648b4b49059fc90bac4fa"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "1cb054aef0a52a91ada0db5274d7731c"
server_key_iv: "51bd1573d53e45dc67f393a7e7850cb6"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.132494 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "16e0fe7138d648b4b49059fc90bac4fa"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "1cb054aef0a52a91ada0db5274d7731c"
server_key_iv: "51bd1573d53e45dc67f393a7e7850cb6"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.136885 8374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:52:39.139675 8394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.140977 8374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:39.141165 8374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "16e0fe7138d648b4b49059fc90bac4fa"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "1cb054aef0a52a91ada0db5274d7731c"
server_key_iv: "51bd1573d53e45dc67f393a7e7850cb6"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.141297 8374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:39.153683 8374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:39.156536 8374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:39.156756 8374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:39.163995 8374 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:41249
I20260809 01:52:39.164404 8468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:41249 every 8 connection(s)
I20260809 01:52:39.165035 8374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:39.167877 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:39.169304 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 8374
I20260809 01:52:39.169400 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:39.169729 3952 external_mini_cluster.cc:1428] Setting key 369a7e84da8f00bb878af1785efd5936
I20260809 01:52:39.174309 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa: Bootstrap starting.
I20260809 01:52:39.176936 8472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:39.177953 8472 log.cc:824] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:39.180014 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa: No bootstrap required, opened a new log
I20260809 01:52:39.183521 8472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0fe7138d648b4b49059fc90bac4fa" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:52:39.183736 8472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:39.183794 8472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e0fe7138d648b4b49059fc90bac4fa, State: Initialized, Role: FOLLOWER
I20260809 01:52:39.184244 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:39.171071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.3.220.62:41249)
0809 01:52:39.171469 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:39.171484 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:39.171519 (+ 35us) server_negotiation.cc:407] Connection header received
0809 01:52:39.172129 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:39.172145 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:39.172469 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:39.172738 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:39.173718 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.174725 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.175908 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.176367 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.176482 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:39.179494 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:39.179499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:39.179512 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:39.179538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:39.181284 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.181814 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.181819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.181823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.181946 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.182240 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.182245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.182251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.182619 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:39.182854 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:39.183111 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 01:52:39.183393 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":129,"threads_started":1}
I20260809 01:52:39.184229 8472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0fe7138d648b4b49059fc90bac4fa" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:52:39.184432 8472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:39.184558 8472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:39.184691 8472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:39.185634 8472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0fe7138d648b4b49059fc90bac4fa" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:52:39.185969 8472 leader_election.cc:302] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e0fe7138d648b4b49059fc90bac4fa; no voters:
I20260809 01:52:39.186321 8472 leader_election.cc:288] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:39.186393 8478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:39.186652 8478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [term 1 LEADER]: Becoming Leader. State: Replica: 16e0fe7138d648b4b49059fc90bac4fa, State: Running, Role: LEADER
I20260809 01:52:39.186968 8478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0fe7138d648b4b49059fc90bac4fa" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:52:39.187650 8472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:39.188686 8481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e0fe7138d648b4b49059fc90bac4fa. Latest consensus state: current_term: 1 leader_uuid: "16e0fe7138d648b4b49059fc90bac4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0fe7138d648b4b49059fc90bac4fa" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } } }
I20260809 01:52:39.188817 8481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:39.189111 8480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16e0fe7138d648b4b49059fc90bac4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0fe7138d648b4b49059fc90bac4fa" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } } }
I20260809 01:52:39.189188 8480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:39.189225 8489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:39.192415 8489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:39.198323 8489 catalog_manager.cc:1347] Generated new cluster ID: bbafeaa3cb5740a694b2a4b327871707
I20260809 01:52:39.198405 8489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:39.212499 8489 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:39.213631 8489 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:39.218681 8489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16e0fe7138d648b4b49059fc90bac4fa: Generated new TSK 0
I20260809 01:52:39.219339 8489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:39.250234 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:39.368994 8507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:39.369232 8507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:39.369282 8507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:39.373255 8507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:39.373317 8507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:39.373373 8507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:39.378587 8507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:39.379664 8507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:39.380707 8507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:39.390868 8515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:39.390830 8518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:39.391052 8507 server_base.cc:1034] running on GCE node
W20260809 01:52:39.390890 8514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:39.391556 8507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:39.392186 8507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:39.393380 8507 hybrid_clock.cc:648] HybridClock initialized: now 1786240359393358 us; error 31 us; skew 500 ppm
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:39.396370 8507 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:39.397159 8507 webserver.cc:460] Webserver started at http://127.3.220.1:34141/ using document root <none> and password file <none>
I20260809 01:52:39.397763 8507 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:39.397827 8507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:39.397997 8507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:39.399937 8507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "a493534949464066ad2e126efc1d76c5"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "07df4edf727f0192dee4569d26e3636d"
server_key_iv: "1e5ded0f2b35c897cc198aa813fe2a10"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.400455 8507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "a493534949464066ad2e126efc1d76c5"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "07df4edf727f0192dee4569d26e3636d"
server_key_iv: "1e5ded0f2b35c897cc198aa813fe2a10"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.404105 8507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 01:52:39.406605 8532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.407908 8507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:39.408043 8507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "a493534949464066ad2e126efc1d76c5"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "07df4edf727f0192dee4569d26e3636d"
server_key_iv: "1e5ded0f2b35c897cc198aa813fe2a10"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.408157 8507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:39.429033 8507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:39.432189 8507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:39.432427 8507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:39.433120 8507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:39.434098 8507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:39.434190 8507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.434271 8507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:39.434332 8507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.444380 8507 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:35653
I20260809 01:52:39.444434 8696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:35653 every 8 connection(s)
I20260809 01:52:39.445427 8507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:39.447108 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 8507
I20260809 01:52:39.447273 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:39.447640 3952 external_mini_cluster.cc:1428] Setting key 2df564f558552bb8f4ce7cb70cc94947
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:52:39.461136 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:39.447338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:40995 (local address 127.3.220.62:41249)
0809 01:52:39.447502 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:39.447509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:39.448891 (+ 1382us) server_negotiation.cc:407] Connection header received
0809 01:52:39.450147 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:39.450154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:39.450222 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:39.450330 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:39.451532 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.452431 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.453695 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.454165 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.454309 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:39.457324 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:39.457327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:39.457329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:39.457359 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:39.459026 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.459536 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.459539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.459540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.459616 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.459983 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.459986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.459987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.460201 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:39.460285 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:39.460898 (+ 613us) server_negotiation.cc:299] Negotiation successful
0809 01:52:39.461010 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:52:39.462031 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:39.447784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.1:40995)
0809 01:52:39.448262 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:39.448315 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:39.449916 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:39.450486 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:39.450493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:39.450873 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:39.451387 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:39.451402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.452573 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.452576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:39.453509 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:39.453519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.454364 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.454368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:39.454532 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:39.455164 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:39.455180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:39.457123 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:39.459152 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:39.459157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:39.459169 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:39.459439 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:39.459744 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:39.459747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:39.459748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:39.459890 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:39.460323 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:39.460328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:39.460550 (+ 222us) client_negotiation.cc:769] Sending connection context
0809 01:52:39.460769 (+ 219us) client_negotiation.cc:240] Negotiation successful
0809 01:52:39.461077 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":142,"threads_started":1}
I20260809 01:52:39.463308 8697 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:52:39.463582 8697 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:39.464093 8697 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:52:39.465855 8415 ts_manager.cc:194] Registered new tserver with Master: a493534949464066ad2e126efc1d76c5 (127.3.220.1:35653)
I20260809 01:52:39.467568 8415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:40995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:39.478749 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:39.591277 8704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:39.591550 8704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:39.591620 8704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:39.595618 8704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:39.595707 8704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:39.595794 8704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:39.601035 8704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:39.602217 8704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:39.603263 8704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:39.613354 8711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:39.613431 8713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:39.613408 8704 server_base.cc:1034] running on GCE node
W20260809 01:52:39.613505 8710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:39.613998 8704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:39.614643 8704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:39.615819 8704 hybrid_clock.cc:648] HybridClock initialized: now 1786240359615786 us; error 54 us; skew 500 ppm
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:39.618774 8704 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:39.619681 8704 webserver.cc:460] Webserver started at http://127.3.220.2:37641/ using document root <none> and password file <none>
I20260809 01:52:39.620234 8704 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:39.620296 8704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:39.620522 8704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:39.622644 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "c6bec3c74dfb4f23a107acc6dab89a40"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "a7ac7bb5372d6020338746861cbba9f2"
server_key_iv: "6cbf1ccfcf827ef3b81bd7a5645811d2"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.623286 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "c6bec3c74dfb4f23a107acc6dab89a40"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "a7ac7bb5372d6020338746861cbba9f2"
server_key_iv: "6cbf1ccfcf827ef3b81bd7a5645811d2"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.627161 8704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:52:39.629766 8730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.631073 8704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:39.631230 8704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "c6bec3c74dfb4f23a107acc6dab89a40"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "a7ac7bb5372d6020338746861cbba9f2"
server_key_iv: "6cbf1ccfcf827ef3b81bd7a5645811d2"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.631346 8704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:39.653431 8704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:39.656917 8704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:39.657133 8704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:39.657776 8704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:39.658607 8704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:39.658684 8704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.658751 8704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:39.658797 8704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.667930 8704 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:34669
I20260809 01:52:39.667958 8881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:34669 every 8 connection(s)
I20260809 01:52:39.668816 8704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:39.675586 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 8704
I20260809 01:52:39.675719 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:39.676002 3952 external_mini_cluster.cc:1428] Setting key 8d86519f1d074a0a19ad6cac369183d8
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:39.683722 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:39.670631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:38319 (local address 127.3.220.62:41249)
0809 01:52:39.670785 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:39.670790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:39.671653 (+ 863us) server_negotiation.cc:407] Connection header received
0809 01:52:39.672976 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:39.672979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:39.673025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:39.673103 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:39.674239 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.674777 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.676338 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.676609 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.676728 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:39.679652 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:39.679662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:39.679664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:39.679690 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:39.681301 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.681976 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.681982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.681984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.682080 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.682494 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.682498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.682500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.682713 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:39.682835 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:39.683391 (+ 556us) server_negotiation.cc:299] Negotiation successful
0809 01:52:39.683529 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 01:52:39.684603 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:39.671063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.2:38319)
0809 01:52:39.671502 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:39.671544 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:39.672768 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:39.673264 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:39.673270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:39.673630 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:39.674099 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:39.674112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.674881 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.674883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:39.676123 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:39.676134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.676782 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.676788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:39.676971 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:39.677695 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:39.677716 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:39.679466 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:39.681445 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:39.681452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:39.681470 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:39.681857 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:39.682208 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:39.682211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:39.682217 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:39.682378 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:39.682834 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:39.682843 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:39.683118 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 01:52:39.683326 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 01:52:39.683703 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260809 01:52:39.686028 8882 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:52:39.686300 8882 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:52:39.686873 8882 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:52:39.688175 8415 ts_manager.cc:194] Registered new tserver with Master: c6bec3c74dfb4f23a107acc6dab89a40 (127.3.220.2:34669)
I20260809 01:52:39.688912 8415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:38319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:39.706450 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:39.822009 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:39.822255 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:39.822299 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:39.826272 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:39.826332 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:39.826388 8892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:39.831596 8892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:39.832623 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:39.833773 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:39.843814 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:39.843829 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:39.843932 8906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:39.844628 8892 server_base.cc:1034] running on GCE node
I20260809 01:52:39.845029 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:39.845820 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:39.847025 8892 hybrid_clock.cc:648] HybridClock initialized: now 1786240359847002 us; error 32 us; skew 500 ppm
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:39.850243 8892 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:39.851153 8892 webserver.cc:460] Webserver started at http://127.3.220.3:41309/ using document root <none> and password file <none>
I20260809 01:52:39.851773 8892 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:39.851845 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:39.852146 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:39.854439 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "a1d95de175ea4c38b3476e5ab24c30c4"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "fb92ead886c298ddc66b3a491e2a5d19"
server_key_iv: "ac5350b5eb6d576df0b1f353b58b3ebe"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.855057 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "a1d95de175ea4c38b3476e5ab24c30c4"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "fb92ead886c298ddc66b3a491e2a5d19"
server_key_iv: "ac5350b5eb6d576df0b1f353b58b3ebe"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.859104 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:52:39.861825 8918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.863164 8892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:39.863305 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "a1d95de175ea4c38b3476e5ab24c30c4"
format_stamp: "Formatted at 2026-08-09 01:52:39 on dist-test-slave-gkw7"
server_key: "fb92ead886c298ddc66b3a491e2a5d19"
server_key_iv: "ac5350b5eb6d576df0b1f353b58b3ebe"
server_key_version: "encryptionkey@0"
I20260809 01:52:39.863421 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:39.879608 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:39.883216 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:39.883450 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:39.884106 8892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:39.884987 8892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:39.885040 8892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.885123 8892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:39.885169 8892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:39.895597 8892 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:42465
I20260809 01:52:39.895613 9071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:42465 every 8 connection(s)
I20260809 01:52:39.896592 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:39.904099 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 8892
I20260809 01:52:39.904242 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:39.904553 3952 external_mini_cluster.cc:1428] Setting key d1b8c0f2ace8b2f7ec41106334007733
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:39.911677 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:39.898668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:43219 (local address 127.3.220.62:41249)
0809 01:52:39.898837 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:39.898843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:39.899741 (+ 898us) server_negotiation.cc:407] Connection header received
0809 01:52:39.901100 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:39.901103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:39.901155 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:39.901238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:39.902506 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.903060 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.904412 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.904691 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.904778 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:39.907659 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:39.907663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:39.907665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:39.907692 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:39.909302 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.909938 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.909943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.909945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.910040 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.910436 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.910440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.910442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.910651 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:39.910767 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:39.911341 (+ 574us) server_negotiation.cc:299] Negotiation successful
0809 01:52:39.911477 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 01:52:39.912529 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:39.899108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.3:43219)
0809 01:52:39.899576 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:39.899619 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:39.900887 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:39.901400 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:39.901406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:39.901874 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:39.902349 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:39.902362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.903183 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.903185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:39.904289 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:39.904303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.904847 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.904851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:39.905027 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:39.905794 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:39.905815 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:39.907459 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:39.909448 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:39.909460 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:39.909476 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:39.909821 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:39.910157 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:39.910160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:39.910162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:39.910321 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:39.910779 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:39.910788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:39.911057 (+ 269us) client_negotiation.cc:769] Sending connection context
0809 01:52:39.911277 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 01:52:39.911610 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260809 01:52:39.913765 9073 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:52:39.913987 9073 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:39.914453 9073 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:52:39.915483 8415 ts_manager.cc:194] Registered new tserver with Master: a1d95de175ea4c38b3476e5ab24c30c4 (127.3.220.3:42465)
I20260809 01:52:39.916082 8415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:43219
I20260809 01:52:39.918766 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 01:52:39 dist-test-slave-gkw7 krb5kdc[8356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240359, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:39.945955 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:39.936935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.3.220.62:41249)
0809 01:52:39.937112 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:39.937117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:39.937309 (+ 192us) server_negotiation.cc:407] Connection header received
0809 01:52:39.937478 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:39.937482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:39.937584 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:39.937680 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:39.938394 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.938924 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.939782 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:39.940008 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:39.940114 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:39.942603 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:39.942606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:39.942609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:39.942640 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:39.943937 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.944370 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.944375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.944376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.944459 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:39.944869 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:39.944871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:39.944872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:39.945081 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:39.945196 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:39.945578 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 01:52:39.945737 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 01:52:39.946429 8415 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43832
I20260809 01:52:39.947453 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 8507
I20260809 01:52:39.959506 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 8704
I20260809 01:52:39.972164 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 8892
I20260809 01:52:39.983642 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 8374
2026-08-09T01:52:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2377 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:40 dist-test-slave-gkw7 krb5kdc[9089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:40 dist-test-slave-gkw7 krb5kdc[9089](info): set up 2 sockets
Aug 09 01:52:40 dist-test-slave-gkw7 krb5kdc[9089](info): commencing operation
krb5kdc: starting...
W20260809 01:52:41.331207 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.319s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260809 01:52:41.490994 3952 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.160s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260809 01:52:41.659242 3952 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.168s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260809 01:52:41.858093 3952 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.199s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/test-user.keytab.
W20260809 01:52:42.313683 3952 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.455s user 0.001s sys 0.001s
Aug 09 01:52:42 dist-test-slave-gkw7 krb5kdc[9089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
W20260809 01:52:42.993791 3952 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.220.62: real 0.655s user 0.000s sys 0.004s
I20260809 01:52:42.994397 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:36025
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:36025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:43.112900 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:43.113142 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:43.113181 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:43.117173 9107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:43.117245 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:43.117265 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:43.117282 9107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:43.117297 9107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:43.122443 9107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:36025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:36025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:43.123433 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:43.124397 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:43.133445 9115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:43.133504 9114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:43.133518 9107 server_base.cc:1034] running on GCE node
W20260809 01:52:43.133751 9118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:43.134326 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:43.135454 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:43.136677 9107 hybrid_clock.cc:648] HybridClock initialized: now 1786240363136632 us; error 45 us; skew 500 ppm
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:43.139506 9107 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:43.140394 9107 webserver.cc:460] Webserver started at http://127.3.220.62:32825/ using document root <none> and password file <none>
I20260809 01:52:43.140973 9107 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:43.141067 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:43.141307 9107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:43.143437 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "78fe3a3a295b4cde889d96acd02a31f5"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "8738d12d046eb296e26b3184a8c96bb1"
server_key_iv: "034b291fb8222be91f865939c8558b96"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.144013 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "78fe3a3a295b4cde889d96acd02a31f5"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "8738d12d046eb296e26b3184a8c96bb1"
server_key_iv: "034b291fb8222be91f865939c8558b96"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.147761 9107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:52:43.150120 9127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.151340 9107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:43.151491 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "78fe3a3a295b4cde889d96acd02a31f5"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "8738d12d046eb296e26b3184a8c96bb1"
server_key_iv: "034b291fb8222be91f865939c8558b96"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.151602 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:43.170307 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:43.173480 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:43.173719 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:43.180824 9107 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:36025
I20260809 01:52:43.180895 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:36025 every 8 connection(s)
I20260809 01:52:43.181856 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:43.184655 9202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:43.190181 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5: Bootstrap starting.
I20260809 01:52:43.192144 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 9107
I20260809 01:52:43.192276 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:43.192575 3952 external_mini_cluster.cc:1428] Setting key ad12fb072e4498bcc8411bae82e3419b
I20260809 01:52:43.192904 9202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:43.193848 9202 log.cc:824] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:43.196077 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5: No bootstrap required, opened a new log
I20260809 01:52:43.200385 9202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe3a3a295b4cde889d96acd02a31f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36025 } }
I20260809 01:52:43.200626 9202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:43.200708 9202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78fe3a3a295b4cde889d96acd02a31f5, State: Initialized, Role: FOLLOWER
I20260809 01:52:43.201140 9202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe3a3a295b4cde889d96acd02a31f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36025 } }
I20260809 01:52:43.201270 9202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:43.201357 9202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:43.201478 9202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:43.202440 9202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe3a3a295b4cde889d96acd02a31f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36025 } }
I20260809 01:52:43.202802 9202 leader_election.cc:302] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78fe3a3a295b4cde889d96acd02a31f5; no voters:
I20260809 01:52:43.203163 9202 leader_election.cc:288] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:43.203244 9210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:43.203493 9210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [term 1 LEADER]: Becoming Leader. State: Replica: 78fe3a3a295b4cde889d96acd02a31f5, State: Running, Role: LEADER
I20260809 01:52:43.203837 9210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe3a3a295b4cde889d96acd02a31f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36025 } }
I20260809 01:52:43.204478 9202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:43.205612 9212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78fe3a3a295b4cde889d96acd02a31f5. Latest consensus state: current_term: 1 leader_uuid: "78fe3a3a295b4cde889d96acd02a31f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe3a3a295b4cde889d96acd02a31f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36025 } } }
I20260809 01:52:43.205749 9212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:43.206259 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78fe3a3a295b4cde889d96acd02a31f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe3a3a295b4cde889d96acd02a31f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36025 } } }
I20260809 01:52:43.206478 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:43.207170 9221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:43.209051 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.193798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.3.220.62:36025)
0809 01:52:43.194256 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:43.194288 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:43.194335 (+ 47us) server_negotiation.cc:407] Connection header received
0809 01:52:43.195104 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:43.195128 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:43.195529 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:43.195903 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:43.196688 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.198242 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.199166 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.199911 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.200075 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.203045 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:43.203051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:43.203063 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:43.203089 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:43.205429 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.205996 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.206002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.206008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.206166 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.206488 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.206492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.206494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.207008 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:43.207261 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:43.208093 (+ 832us) server_negotiation.cc:299] Negotiation successful
0809 01:52:43.208335 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
I20260809 01:52:43.210714 9221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:43.216203 9221 catalog_manager.cc:1347] Generated new cluster ID: 3f3227e2a9f74ddfb12bc47692f414b6
I20260809 01:52:43.216274 9221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:43.231489 9221 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:43.232722 9221 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:43.242095 9221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78fe3a3a295b4cde889d96acd02a31f5: Generated new TSK 0
I20260809 01:52:43.242802 9221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:43.280438 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36025
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:43.392032 9238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:43.392284 9238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:43.392325 9238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:43.396498 9238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:43.396552 9238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:43.396663 9238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:43.401922 9238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:43.402972 9238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:43.404059 9238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:43.414038 9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:43.414064 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:43.414153 9238 server_base.cc:1034] running on GCE node
W20260809 01:52:43.414099 9248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:43.414708 9238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:43.415356 9238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:43.416517 9238 hybrid_clock.cc:648] HybridClock initialized: now 1786240363416481 us; error 47 us; skew 500 ppm
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:43.419309 9238 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:43.420164 9238 webserver.cc:460] Webserver started at http://127.3.220.1:41373/ using document root <none> and password file <none>
I20260809 01:52:43.420720 9238 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:43.420818 9238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:43.421066 9238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:43.423210 9238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "bef24033286746e699879ee15d9074fe"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "d6acbdc37f7a3fc4306d2bf65c4556fe"
server_key_iv: "4f48e29dd5353f3d08febada91da4240"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.423805 9238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "bef24033286746e699879ee15d9074fe"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "d6acbdc37f7a3fc4306d2bf65c4556fe"
server_key_iv: "4f48e29dd5353f3d08febada91da4240"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.427647 9238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 01:52:43.430127 9257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.431367 9238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:43.431495 9238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "bef24033286746e699879ee15d9074fe"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "d6acbdc37f7a3fc4306d2bf65c4556fe"
server_key_iv: "4f48e29dd5353f3d08febada91da4240"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.431607 9238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:43.447563 9238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:43.450587 9238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:43.450804 9238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:43.451422 9238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:43.452332 9238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:43.452406 9238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.452481 9238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:43.452526 9238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.461661 9238 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:37331
I20260809 01:52:43.461733 9428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:37331 every 8 connection(s)
I20260809 01:52:43.462603 9238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:43.467491 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 9238
I20260809 01:52:43.467622 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:43.467934 3952 external_mini_cluster.cc:1428] Setting key fc8697e9555015ee1a4701dc766f7cd4
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:43.477188 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.464509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:52533 (local address 127.3.220.62:36025)
0809 01:52:43.464648 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:43.464651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:43.465395 (+ 744us) server_negotiation.cc:407] Connection header received
0809 01:52:43.466778 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:43.466781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:43.466835 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:43.466915 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:43.468403 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.469044 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.470335 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.470749 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.470875 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.473511 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:43.473515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:43.473531 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:43.473579 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:43.475148 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.475643 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.475648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.475650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.475733 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.476085 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.476091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.476094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.476284 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:43.476394 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:43.476870 (+ 476us) server_negotiation.cc:299] Negotiation successful
0809 01:52:43.477004 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:52:43.477998 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.464809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:36025 (local address 127.3.220.1:52533)
0809 01:52:43.465240 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:43.465282 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:43.466573 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:43.467078 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:43.467085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:43.467575 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:43.468237 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:43.468257 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.469186 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.469189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:43.470205 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:43.470215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.470898 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.470901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:43.471041 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.471672 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:43.471688 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:43.473353 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:43.475271 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:43.475277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:43.475290 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:43.475539 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:43.475825 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:43.475829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:43.475831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:43.475986 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:43.476395 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:43.476402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:43.476633 (+ 231us) client_negotiation.cc:769] Sending connection context
0809 01:52:43.476827 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 01:52:43.477133 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:52:43.479130 9429 heartbeater.cc:347] Connected to a master server at 127.3.220.62:36025
I20260809 01:52:43.479373 9429 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:43.479897 9429 heartbeater.cc:511] Master 127.3.220.62:36025 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:52:43.481583 9154 ts_manager.cc:194] Registered new tserver with Master: bef24033286746e699879ee15d9074fe (127.3.220.1:37331)
I20260809 01:52:43.482720 9154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:52533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:43.499033 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36025
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:43.612571 9437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:43.612813 9437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:43.612856 9437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:43.616941 9437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:43.616994 9437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:43.617051 9437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:43.622287 9437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:43.623243 9437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:43.624253 9437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:52:43.634196 9437 server_base.cc:1034] running on GCE node
W20260809 01:52:43.634158 9448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:43.634164 9444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:43.634158 9446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:43.634838 9437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:43.635560 9437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:43.636799 9437 hybrid_clock.cc:648] HybridClock initialized: now 1786240363636765 us; error 48 us; skew 500 ppm
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:43.639762 9437 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:43.640650 9437 webserver.cc:460] Webserver started at http://127.3.220.2:46183/ using document root <none> and password file <none>
I20260809 01:52:43.641225 9437 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:43.641330 9437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:43.641642 9437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:43.643746 9437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "859ccd57fdd6421c8751e7202e31c654"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "85f68df4ebd69f4e8b49c021deffaa4b"
server_key_iv: "12b1e483feda45752696ad8ad3eece73"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.644352 9437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "859ccd57fdd6421c8751e7202e31c654"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "85f68df4ebd69f4e8b49c021deffaa4b"
server_key_iv: "12b1e483feda45752696ad8ad3eece73"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.648283 9437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 01:52:43.650903 9457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.652096 9437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:43.652241 9437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "859ccd57fdd6421c8751e7202e31c654"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "85f68df4ebd69f4e8b49c021deffaa4b"
server_key_iv: "12b1e483feda45752696ad8ad3eece73"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.652369 9437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:43.670389 9437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:43.673810 9437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:43.674033 9437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:43.674636 9437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:43.675562 9437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:43.675637 9437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.675709 9437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:43.675745 9437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.685329 9437 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:44393
I20260809 01:52:43.685354 9624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:44393 every 8 connection(s)
I20260809 01:52:43.686260 9437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:43.696305 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 9437
I20260809 01:52:43.696460 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:43.696772 3952 external_mini_cluster.cc:1428] Setting key afdca7dec1fcb564a163ea0bf4d58061
I20260809 01:52:43.700376 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.688092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:54167 (local address 127.3.220.62:36025)
0809 01:52:43.688294 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:43.688298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:43.689124 (+ 826us) server_negotiation.cc:407] Connection header received
0809 01:52:43.690588 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:43.690591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:43.690639 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:43.690718 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:43.691870 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.692397 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.693477 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.693744 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.693824 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.696344 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:43.696348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:43.696349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:43.696394 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:43.697982 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.698618 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.698622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.698624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.698719 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.699060 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.699066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.699071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.699387 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:43.699496 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:43.699976 (+ 480us) server_negotiation.cc:299] Negotiation successful
0809 01:52:43.700127 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260809 01:52:43.701210 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.688480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:36025 (local address 127.3.220.2:54167)
0809 01:52:43.688961 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:43.689005 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:43.690392 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:43.690879 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:43.690886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:43.691271 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:43.691740 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:43.691754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.692506 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.692508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:43.693373 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:43.693382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.693875 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.693878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:43.694017 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.694541 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:43.694556 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:43.696164 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:43.698150 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:43.698157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:43.698173 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:43.698510 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:43.698824 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:43.698826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:43.698827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:43.698962 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:43.699494 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:43.699499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:43.699727 (+ 228us) client_negotiation.cc:769] Sending connection context
0809 01:52:43.699893 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 01:52:43.700150 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":103,"threads_started":1}
I20260809 01:52:43.702499 9625 heartbeater.cc:347] Connected to a master server at 127.3.220.62:36025
I20260809 01:52:43.702768 9625 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:43.703269 9625 heartbeater.cc:511] Master 127.3.220.62:36025 requested a full tablet report, sending...
I20260809 01:52:43.704432 9154 ts_manager.cc:194] Registered new tserver with Master: 859ccd57fdd6421c8751e7202e31c654 (127.3.220.2:44393)
I20260809 01:52:43.705034 9154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:54167
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:43.729473 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36025
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:43.843343 9632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:43.843626 9632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:43.843704 9632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:43.847712 9632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:43.847806 9632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:43.847898 9632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:43.853020 9632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:43.854089 9632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:43.855194 9632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:43.865260 9643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:43.865288 9640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:43.865486 9632 server_base.cc:1034] running on GCE node
W20260809 01:52:43.865407 9641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:43.865880 9632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:43.866478 9632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:43.867636 9632 hybrid_clock.cc:648] HybridClock initialized: now 1786240363867608 us; error 38 us; skew 500 ppm
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:43.870441 9632 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:43.871313 9632 webserver.cc:460] Webserver started at http://127.3.220.3:42093/ using document root <none> and password file <none>
I20260809 01:52:43.871873 9632 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:43.871937 9632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:43.872174 9632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:43.874259 9632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "3f501a7a18044429a302f343551db818"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "1861b1b47a261a3a261c105413a1ef82"
server_key_iv: "8145b20d455f99a07e2d36c5478b1f30"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.874838 9632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "3f501a7a18044429a302f343551db818"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "1861b1b47a261a3a261c105413a1ef82"
server_key_iv: "8145b20d455f99a07e2d36c5478b1f30"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.878674 9632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 01:52:43.881084 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.882391 9632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:52:43.882532 9632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "3f501a7a18044429a302f343551db818"
format_stamp: "Formatted at 2026-08-09 01:52:43 on dist-test-slave-gkw7"
server_key: "1861b1b47a261a3a261c105413a1ef82"
server_key_iv: "8145b20d455f99a07e2d36c5478b1f30"
server_key_version: "encryptionkey@0"
I20260809 01:52:43.882647 9632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:43.896586 9632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:43.899654 9632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:43.899873 9632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:43.900468 9632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:43.901300 9632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:43.901373 9632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.901448 9632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:43.901502 9632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:43.910614 9632 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:43523
I20260809 01:52:43.910646 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:43523 every 8 connection(s)
I20260809 01:52:43.911494 9632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:43.916384 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 9632
I20260809 01:52:43.916538 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:43.916844 3952 external_mini_cluster.cc:1428] Setting key 324b9b9e500c30100c363a7e398bc5a8
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:43.925279 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.913251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:43331 (local address 127.3.220.62:36025)
0809 01:52:43.913388 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:43.913392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:43.914232 (+ 840us) server_negotiation.cc:407] Connection header received
0809 01:52:43.915614 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:43.915617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:43.915674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:43.915762 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:43.917304 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.917972 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.919205 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.919422 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.919533 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.921984 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:43.921992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:43.921994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:43.922018 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:43.923259 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.923771 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.923779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.923780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.923854 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.924194 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.924197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.924199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.924373 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:43.924483 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:43.924965 (+ 482us) server_negotiation.cc:299] Negotiation successful
0809 01:52:43.925084 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:52:43.926200 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.913635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:36025 (local address 127.3.220.3:43331)
0809 01:52:43.914067 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:43.914108 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:43.915399 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:43.915944 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:43.915951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:43.916452 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:43.917136 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:43.917167 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.918087 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.918091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:43.919082 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:43.919092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.919569 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.919574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:43.919753 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.920350 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:43.920365 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:43.921810 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:43.923379 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:43.923385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:43.923399 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:43.923650 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:43.923946 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:43.923950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:43.923952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:43.924089 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:43.924481 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:43.924489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:43.924731 (+ 242us) client_negotiation.cc:769] Sending connection context
0809 01:52:43.924923 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 01:52:43.925238 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260809 01:52:43.927243 9814 heartbeater.cc:347] Connected to a master server at 127.3.220.62:36025
I20260809 01:52:43.927493 9814 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:43.927970 9814 heartbeater.cc:511] Master 127.3.220.62:36025 requested a full tablet report, sending...
I20260809 01:52:43.928967 9154 ts_manager.cc:194] Registered new tserver with Master: 3f501a7a18044429a302f343551db818 (127.3.220.3:43523)
I20260809 01:52:43.929688 9154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:43331
I20260809 01:52:43.930863 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:43.963176 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.952902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.3.220.1:37331)
0809 01:52:43.953227 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:43.953243 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:43.953272 (+ 29us) server_negotiation.cc:407] Connection header received
0809 01:52:43.953327 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:43.953333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:43.953521 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:43.953735 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:43.954647 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.955614 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.956645 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.957039 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.957157 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.959732 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:43.959750 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:43.959764 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:43.959789 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:43.961229 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.961733 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.961740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.961742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.961833 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.962124 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.962129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.962130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.962394 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:43.962662 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:43.962817 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 01:52:43.962977 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":113,"threads_started":1}
Aug 09 01:52:43 dist-test-slave-gkw7 krb5kdc[9089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:43.974325 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:43.966196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.3.220.62:36025)
0809 01:52:43.966343 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:43.966350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:43.966366 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:52:43.966524 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:43.966527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:43.966581 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:43.966664 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:43.967252 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.967775 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.968547 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:43.968780 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:43.968884 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:43.971291 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:43.971295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:43.971298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:43.971330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:43.972543 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.972966 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.972969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.972971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.973053 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:43.973327 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:43.973331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:43.973333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:43.973636 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:43.973782 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:43.974025 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 01:52:43.974146 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 01:52:43.974764 9154 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40868
I20260809 01:52:43.975459 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 9238
I20260809 01:52:43.997936 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 9437
I20260809 01:52:44.009050 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 9632
I20260809 01:52:44.022424 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 9107
2026-08-09T01:52:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4036 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:44 dist-test-slave-gkw7 krb5kdc[9829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:44 dist-test-slave-gkw7 krb5kdc[9829](info): set up 2 sockets
Aug 09 01:52:44 dist-test-slave-gkw7 krb5kdc[9829](info): commencing operation
krb5kdc: starting...
W20260809 01:52:45.299937 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:45 dist-test-slave-gkw7 krb5kdc[9829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:45.432047 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35069
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:35069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:45.545179 9848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:45.545483 9848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:45.545581 9848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:45.549695 9848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:45.549770 9848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:45.549808 9848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:45.549860 9848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:45.549898 9848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:45.555083 9848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:35069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:45.556110 9848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:45.557165 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:45.566103 9859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:45.566236 9848 server_base.cc:1034] running on GCE node
W20260809 01:52:45.566109 9856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:45.566109 9855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:45.567106 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:45.568197 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:45.569432 9848 hybrid_clock.cc:648] HybridClock initialized: now 1786240365569398 us; error 42 us; skew 500 ppm
Aug 09 01:52:45 dist-test-slave-gkw7 krb5kdc[9829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240365, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:45.577164 9848 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:45.578042 9848 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:45.578116 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:45.578302 9848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:45.580469 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "fc2847544c164608818957cc383322e6"
format_stamp: "Formatted at 2026-08-09 01:52:45 on dist-test-slave-gkw7"
server_key: "835039cf45d2829d0e7c8ec8e4d9555a"
server_key_iv: "aef8ae72b72038408cbfbce4281fbf89"
server_key_version: "encryptionkey@0"
I20260809 01:52:45.581035 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "fc2847544c164608818957cc383322e6"
format_stamp: "Formatted at 2026-08-09 01:52:45 on dist-test-slave-gkw7"
server_key: "835039cf45d2829d0e7c8ec8e4d9555a"
server_key_iv: "aef8ae72b72038408cbfbce4281fbf89"
server_key_version: "encryptionkey@0"
I20260809 01:52:45.585208 9848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 01:52:45.588068 9868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:45.589619 9848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:45.589754 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "fc2847544c164608818957cc383322e6"
format_stamp: "Formatted at 2026-08-09 01:52:45 on dist-test-slave-gkw7"
server_key: "835039cf45d2829d0e7c8ec8e4d9555a"
server_key_iv: "aef8ae72b72038408cbfbce4281fbf89"
server_key_version: "encryptionkey@0"
I20260809 01:52:45.589854 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:45.624773 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:45.628567 9848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:45.628757 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:45.636204 9848 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:35069
I20260809 01:52:45.636490 9939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:35069 every 8 connection(s)
I20260809 01:52:45.636956 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:45.638994 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 9848
I20260809 01:52:45.639128 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:45.639438 3952 external_mini_cluster.cc:1428] Setting key a97a13e56ff8a8b72456a4e2cef37f70
I20260809 01:52:45.640492 9941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:45.647226 9941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6: Bootstrap starting.
Aug 09 01:52:45 dist-test-slave-gkw7 krb5kdc[9829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:45.650019 9941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:45.650805 9941 log.cc:824] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:45.652724 9941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6: No bootstrap required, opened a new log
I20260809 01:52:45.654516 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:45.640927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.3.220.62:35069)
0809 01:52:45.641378 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:45.641394 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:45.641430 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:52:45.642119 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:45.642135 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:45.642471 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:45.642757 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:45.643716 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:45.644779 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:45.646017 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:45.646495 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:45.646635 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:45.649502 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:45.649508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:45.649524 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:45.649593 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:45.651551 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:45.652044 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:45.652050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:45.652055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:45.652178 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:45.652467 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:45.652471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:45.652473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:45.652926 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:45.653138 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:45.653438 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 01:52:45.653817 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":157,"threads_started":1}
I20260809 01:52:45.656613 9941 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2847544c164608818957cc383322e6" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35069 } }
I20260809 01:52:45.656915 9941 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:45.657008 9941 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2847544c164608818957cc383322e6, State: Initialized, Role: FOLLOWER
I20260809 01:52:45.657410 9941 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2847544c164608818957cc383322e6" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35069 } }
I20260809 01:52:45.657565 9941 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:45.657634 9941 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:45.657708 9941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:45.658651 9941 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2847544c164608818957cc383322e6" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35069 } }
I20260809 01:52:45.658949 9941 leader_election.cc:302] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc2847544c164608818957cc383322e6; no voters:
I20260809 01:52:45.659246 9941 leader_election.cc:288] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:45.659376 9946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:45.659646 9946 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [term 1 LEADER]: Becoming Leader. State: Replica: fc2847544c164608818957cc383322e6, State: Running, Role: LEADER
I20260809 01:52:45.659974 9946 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2847544c164608818957cc383322e6" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35069 } }
I20260809 01:52:45.660495 9941 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:45.661782 9948 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc2847544c164608818957cc383322e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2847544c164608818957cc383322e6" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35069 } } }
I20260809 01:52:45.661942 9948 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:45.662338 9964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:45.662307 9951 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc2847544c164608818957cc383322e6. Latest consensus state: current_term: 1 leader_uuid: "fc2847544c164608818957cc383322e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2847544c164608818957cc383322e6" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35069 } } }
I20260809 01:52:45.662467 9951 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:45.665582 9964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:45.671332 9964 catalog_manager.cc:1347] Generated new cluster ID: fff52457c7414517861ab0ecfe684fc9
I20260809 01:52:45.671415 9964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:45.690960 9964 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:45.692261 9964 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:45.701674 9964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6: Generated new TSK 0
I20260809 01:52:45.702584 9964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:45.741451 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35069
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:45.862635 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:45.862907 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:45.862954 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:45.866983 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:45.867039 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:45.867146 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:45.872387 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:45.873452 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:45.874521 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:45.884620 9990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:45.884596 9994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:45.884620 9981 server_base.cc:1034] running on GCE node
W20260809 01:52:45.884635 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:45.885306 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:45.885999 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:45.887205 9981 hybrid_clock.cc:648] HybridClock initialized: now 1786240365887189 us; error 35 us; skew 500 ppm
Aug 09 01:52:45 dist-test-slave-gkw7 krb5kdc[9829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240365, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:45.890275 9981 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:45.891005 9981 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:45.891068 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:45.891235 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:45.893381 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "30251326617c485a99204ab03bae5fe3"
format_stamp: "Formatted at 2026-08-09 01:52:45 on dist-test-slave-gkw7"
server_key: "2cafc9dc6388f87b2854305d92008c02"
server_key_iv: "ec4c09dd311a4bebdd02136e635b2026"
server_key_version: "encryptionkey@0"
I20260809 01:52:45.894052 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "30251326617c485a99204ab03bae5fe3"
format_stamp: "Formatted at 2026-08-09 01:52:45 on dist-test-slave-gkw7"
server_key: "2cafc9dc6388f87b2854305d92008c02"
server_key_iv: "ec4c09dd311a4bebdd02136e635b2026"
server_key_version: "encryptionkey@0"
I20260809 01:52:45.897989 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 01:52:45.900420 10001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:45.901698 9981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:45.901830 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "30251326617c485a99204ab03bae5fe3"
format_stamp: "Formatted at 2026-08-09 01:52:45 on dist-test-slave-gkw7"
server_key: "2cafc9dc6388f87b2854305d92008c02"
server_key_iv: "ec4c09dd311a4bebdd02136e635b2026"
server_key_version: "encryptionkey@0"
I20260809 01:52:45.901950 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:45.915077 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:45.917975 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:45.918179 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:45.918833 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:45.919682 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:45.919730 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:45.919821 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:45.919857 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:45.929059 9981 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:36153
I20260809 01:52:45.929082 10163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:36153 every 8 connection(s)
I20260809 01:52:45.929744 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:45.939253 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 9981
Aug 09 01:52:45 dist-test-slave-gkw7 krb5kdc[9829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240365, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:45.939404 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:45.939725 3952 external_mini_cluster.cc:1428] Setting key 0685e3f649a2d251027e1a77b82aa628
I20260809 01:52:45.943909 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:45.931580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:45609 (local address 127.3.220.62:35069)
0809 01:52:45.931737 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:45.931741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:45.932926 (+ 1185us) server_negotiation.cc:407] Connection header received
0809 01:52:45.933887 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:45.933890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:45.933950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:45.934058 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:45.935277 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:45.935819 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:45.936971 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:45.937226 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:45.937346 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:45.939999 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:45.940003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:45.940006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:45.940035 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:45.941775 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:45.942283 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:45.942289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:45.942294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:45.942393 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:45.942725 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:45.942731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:45.942736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:45.942987 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:45.943099 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:45.943596 (+ 497us) server_negotiation.cc:299] Negotiation successful
0809 01:52:45.943734 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 01:52:45.944717 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:45.931968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35069 (local address 127.3.220.1:45609)
0809 01:52:45.932765 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:45.932808 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:45.933655 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:45.934242 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:45.934249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:45.934614 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:45.935098 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:45.935111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:45.935944 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:45.935948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:45.936837 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:45.936847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:45.937389 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:45.937393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:45.937591 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:45.938214 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:45.938229 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:45.939814 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:45.941913 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:45.941918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:45.941931 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:45.942182 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:45.942496 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:45.942498 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:45.942500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:45.942634 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:45.943091 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:45.943095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:45.943332 (+ 237us) client_negotiation.cc:769] Sending connection context
0809 01:52:45.943529 (+ 197us) client_negotiation.cc:240] Negotiation successful
0809 01:52:45.943879 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260809 01:52:45.946110 10165 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35069
I20260809 01:52:45.946372 10165 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:45.946835 10165 heartbeater.cc:511] Master 127.3.220.62:35069 requested a full tablet report, sending...
I20260809 01:52:45.948598 9890 ts_manager.cc:194] Registered new tserver with Master: 30251326617c485a99204ab03bae5fe3 (127.3.220.1:36153)
I20260809 01:52:45.950299 9890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:45609
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:45.972652 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35069
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:46.088824 10173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:46.089205 10173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:46.089286 10173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:46.093467 10173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:46.093587 10173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:46.093696 10173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:46.099254 10173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:46.100399 10173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:46.101486 10173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:46.111718 10183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:46.111682 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:46.111927 10173 server_base.cc:1034] running on GCE node
W20260809 01:52:46.112033 10188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:46.112514 10173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:46.113184 10173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:46.114374 10173 hybrid_clock.cc:648] HybridClock initialized: now 1786240366114359 us; error 28 us; skew 500 ppm
Aug 09 01:52:46 dist-test-slave-gkw7 krb5kdc[9829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:46.117292 10173 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:46.118093 10173 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:46.118170 10173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:46.118345 10173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:46.120397 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "08b45d3a12de440d8552941b9eaacd74"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "074fd241fdd4d94d998f04f5620a8a35"
server_key_iv: "45591ceb8cbab5b7278a78d169e5b3e3"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.120925 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "08b45d3a12de440d8552941b9eaacd74"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "074fd241fdd4d94d998f04f5620a8a35"
server_key_iv: "45591ceb8cbab5b7278a78d169e5b3e3"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.124784 10173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:52:46.127303 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:46.128551 10173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:46.128649 10173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "08b45d3a12de440d8552941b9eaacd74"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "074fd241fdd4d94d998f04f5620a8a35"
server_key_iv: "45591ceb8cbab5b7278a78d169e5b3e3"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.128737 10173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:46.143671 10173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:46.146642 10173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:46.146801 10173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:46.147333 10173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:46.148141 10173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:46.148196 10173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:46.148237 10173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:46.148259 10173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:46.157147 10173 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:44977
I20260809 01:52:46.157181 10351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:44977 every 8 connection(s)
I20260809 01:52:46.157899 10173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:46.160362 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 10173
I20260809 01:52:46.160482 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:46.160808 3952 external_mini_cluster.cc:1428] Setting key 2d65f86bd7fef367b3a52edf4820a01f
Aug 09 01:52:46 dist-test-slave-gkw7 krb5kdc[9829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:52:46.174165 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.159792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:60557 (local address 127.3.220.62:35069)
0809 01:52:46.159961 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.159966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.161497 (+ 1531us) server_negotiation.cc:407] Connection header received
0809 01:52:46.162639 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.162642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.162691 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.162775 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:46.164337 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.165103 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.166492 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.166920 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.167047 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.169904 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:46.169912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:46.169918 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:46.169954 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:46.171736 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:46.172330 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:46.172335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:46.172337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:46.172433 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:46.172820 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:46.172825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:46.172827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:46.173096 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:46.173207 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.173914 (+ 707us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.174035 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 01:52:46.175221 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.160241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35069 (local address 127.3.220.2:60557)
0809 01:52:46.161325 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.161380 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:46.162433 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:46.162925 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:46.162935 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:46.163474 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:46.164187 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.164210 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.165244 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.165248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.166328 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.166342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.167150 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.167215 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.167428 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.168073 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:46.168090 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:46.169714 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:46.171867 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:46.171872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:46.171886 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:46.172195 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:46.172540 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:46.172542 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:46.172543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:46.172688 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:46.173284 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:46.173291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:46.173628 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 01:52:46.173811 (+ 183us) client_negotiation.cc:240] Negotiation successful
0809 01:52:46.174203 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:52:46.176592 10354 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35069
I20260809 01:52:46.176856 10354 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:46.177366 10354 heartbeater.cc:511] Master 127.3.220.62:35069 requested a full tablet report, sending...
I20260809 01:52:46.178580 9890 ts_manager.cc:194] Registered new tserver with Master: 08b45d3a12de440d8552941b9eaacd74 (127.3.220.2:44977)
I20260809 01:52:46.179183 9890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:60557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:46.194105 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35069
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:46.315532 10362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:46.315824 10362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:46.315879 10362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:46.320122 10362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:46.320182 10362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:46.320242 10362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:46.325613 10362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:46.326673 10362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:46.327687 10362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:46.337481 10370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:46.337669 10362 server_base.cc:1034] running on GCE node
W20260809 01:52:46.337496 10374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:46.337486 10369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:46.338189 10362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:46.338876 10362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:46.340070 10362 hybrid_clock.cc:648] HybridClock initialized: now 1786240366340049 us; error 31 us; skew 500 ppm
Aug 09 01:52:46 dist-test-slave-gkw7 krb5kdc[9829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:46.343008 10362 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:46.343837 10362 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:46.343935 10362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:46.344184 10362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:46.346382 10362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "091aa27fc3b245f8949b5d1a7a002120"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "19cbf4fe3b529a2326249074bdd29807"
server_key_iv: "b122830b76796587e5a9bd681c369f49"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.346978 10362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "091aa27fc3b245f8949b5d1a7a002120"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "19cbf4fe3b529a2326249074bdd29807"
server_key_iv: "b122830b76796587e5a9bd681c369f49"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.350790 10362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:52:46.353351 10388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:46.354754 10362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:46.354908 10362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "091aa27fc3b245f8949b5d1a7a002120"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "19cbf4fe3b529a2326249074bdd29807"
server_key_iv: "b122830b76796587e5a9bd681c369f49"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.355028 10362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:46.369493 10362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:46.372651 10362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:46.372875 10362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:46.373515 10362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:46.374455 10362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:46.374531 10362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:46.374601 10362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:46.374658 10362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:46.384019 10362 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:37921
I20260809 01:52:46.384037 10554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:37921 every 8 connection(s)
I20260809 01:52:46.384671 10362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:46.392354 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 10362
I20260809 01:52:46.392496 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:46.392871 3952 external_mini_cluster.cc:1428] Setting key 33e1ded41178b0090c0eba5e97f8b22d
Aug 09 01:52:46 dist-test-slave-gkw7 krb5kdc[9829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:46.398744 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.386620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:56171 (local address 127.3.220.62:35069)
0809 01:52:46.386772 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.386776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.387610 (+ 834us) server_negotiation.cc:407] Connection header received
0809 01:52:46.389008 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.389011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.389062 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.389147 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:46.390378 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.390934 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.392054 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.392276 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.392359 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.395552 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:46.395555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:46.395557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:46.395582 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:46.396779 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:46.397247 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:46.397249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:46.397250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:46.397328 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:46.397739 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:46.397742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:46.397743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:46.397937 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:46.398016 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.398518 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.398623 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 01:52:46.399564 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.386955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35069 (local address 127.3.220.3:56171)
0809 01:52:46.387435 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.387478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:46.388783 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:46.389338 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:46.389345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:46.389731 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:46.390213 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.390226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.391045 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.391047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.391940 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.391949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.392446 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.392452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.392686 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.393444 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:46.393461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:46.395380 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:46.396880 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:46.396885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:46.396897 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:46.397155 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:46.397426 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:46.397428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:46.397429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:46.397607 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:46.398033 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:46.398038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:46.398296 (+ 258us) client_negotiation.cc:769] Sending connection context
0809 01:52:46.398452 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 01:52:46.398721 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20260809 01:52:46.400740 10556 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35069
I20260809 01:52:46.400998 10556 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:46.401464 10556 heartbeater.cc:511] Master 127.3.220.62:35069 requested a full tablet report, sending...
I20260809 01:52:46.402539 9890 ts_manager.cc:194] Registered new tserver with Master: 091aa27fc3b245f8949b5d1a7a002120 (127.3.220.3:37921)
I20260809 01:52:46.403141 9890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:56171
I20260809 01:52:46.407181 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:52:46.416960 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.409090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.3.220.62:35069)
0809 01:52:46.409296 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.409301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.409433 (+ 132us) server_negotiation.cc:407] Connection header received
0809 01:52:46.409571 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.409577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.409621 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.409684 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:46.410359 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.410851 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.411695 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.411976 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.412122 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.413388 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:46.413392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:46.413394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:46.413418 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:46.415270 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:46.415653 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:46.415656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:46.415657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:46.415748 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:46.415989 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:46.415992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:46.415993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:46.416277 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:46.416404 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.416582 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.416720 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260809 01:52:46.419872 9890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:52:46.422956 9890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:46.438006 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.431695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:37921 (local address 127.0.0.1:35306)
0809 01:52:46.432631 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.432646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:46.432737 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:46.433969 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:46.433972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:46.433987 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:46.434073 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.434081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.435164 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.435169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.436459 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.436469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.437414 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.437417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.437498 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.437582 (+ 84us) client_negotiation.cc:769] Sending connection context
0809 01:52:46.437701 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 01:52:46.437844 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":223,"thread_start_us":95,"threads_started":1}
I20260809 01:52:46.438037 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.432415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.3.220.3:37921)
0809 01:52:46.433493 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.433508 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.433534 (+ 26us) server_negotiation.cc:407] Connection header received
0809 01:52:46.433637 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.433640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.433814 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.434206 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:46.434334 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.435034 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.436586 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.437288 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.437395 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.437565 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.437749 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.437853 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":124,"threads_started":1}
I20260809 01:52:46.438642 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.432000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:36153 (local address 127.0.0.1:41404)
0809 01:52:46.432928 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.432943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:46.433075 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:46.433823 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:46.433828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:46.433845 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:46.433974 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.433981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.435468 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.435472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.436902 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.436914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.438204 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.438208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.438301 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.438324 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 01:52:46.438427 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 01:52:46.438503 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":124,"thread_start_us":97,"threads_started":1}
I20260809 01:52:46.438998 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.431687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:44977 (local address 127.0.0.1:60560)
0809 01:52:46.432411 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.432464 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:46.432607 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:46.433625 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:46.433631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:46.433654 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:46.433835 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.433842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.435468 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.435472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.436903 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.436914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.438614 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.438618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.438720 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.438747 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:52:46.438802 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 01:52:46.438875 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":175,"thread_start_us":88,"threads_started":1}
I20260809 01:52:46.439361 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.432972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.3.220.1:36153)
0809 01:52:46.433351 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.433357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.433378 (+ 21us) server_negotiation.cc:407] Connection header received
0809 01:52:46.433452 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.433457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.433677 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.433835 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:46.434099 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.435311 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.437050 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.437890 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.437994 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.438134 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.438463 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.439171 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":97,"threads_started":1}
I20260809 01:52:46.439829 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.431881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.3.220.2:44977)
0809 01:52:46.432991 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.433005 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.433031 (+ 26us) server_negotiation.cc:407] Connection header received
0809 01:52:46.433114 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.433119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.433416 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.433982 (+ 566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:46.434166 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.435309 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.437641 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.438466 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.439416 (+ 950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.439572 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.439621 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.439703 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1003,"thread_start_us":122,"threads_started":1}
I20260809 01:52:46.441898 10453 tablet_service.cc:1467] Processing CreateTablet for tablet dc67b5744b0c410b8706cd8ee74c60d1 (DEFAULT_TABLE table=test-table [id=916f56c484564302b63e8c68d5ad9bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:46.442140 10068 tablet_service.cc:1467] Processing CreateTablet for tablet dc67b5744b0c410b8706cd8ee74c60d1 (DEFAULT_TABLE table=test-table [id=916f56c484564302b63e8c68d5ad9bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:46.442464 10260 tablet_service.cc:1467] Processing CreateTablet for tablet dc67b5744b0c410b8706cd8ee74c60d1 (DEFAULT_TABLE table=test-table [id=916f56c484564302b63e8c68d5ad9bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:46.443097 10453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67b5744b0c410b8706cd8ee74c60d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:46.443173 10068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67b5744b0c410b8706cd8ee74c60d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:46.443379 10260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc67b5744b0c410b8706cd8ee74c60d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:46.449520 10579 tablet_bootstrap.cc:492] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3: Bootstrap starting.
I20260809 01:52:46.450187 10580 tablet_bootstrap.cc:492] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74: Bootstrap starting.
I20260809 01:52:46.450628 10582 tablet_bootstrap.cc:492] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: Bootstrap starting.
I20260809 01:52:46.452059 10579 tablet_bootstrap.cc:654] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:46.452710 10582 tablet_bootstrap.cc:654] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:46.453027 10579 log.cc:824] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:46.453963 10582 log.cc:824] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:46.455238 10579 tablet_bootstrap.cc:492] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3: No bootstrap required, opened a new log
I20260809 01:52:46.455453 10579 ts_tablet_manager.cc:1397] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:52:46.455588 10580 tablet_bootstrap.cc:654] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:46.456099 10582 tablet_bootstrap.cc:492] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: No bootstrap required, opened a new log
I20260809 01:52:46.456288 10582 ts_tablet_manager.cc:1397] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:52:46.456710 10580 log.cc:824] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:46.458819 10580 tablet_bootstrap.cc:492] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74: No bootstrap required, opened a new log
I20260809 01:52:46.459002 10580 ts_tablet_manager.cc:1397] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260809 01:52:46.459426 10579 raft_consensus.cc:348] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.459960 10579 raft_consensus.cc:374] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:46.460110 10579 raft_consensus.cc:729] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30251326617c485a99204ab03bae5fe3, State: Initialized, Role: FOLLOWER
I20260809 01:52:46.460099 10582 raft_consensus.cc:348] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.460407 10582 raft_consensus.cc:374] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:46.460498 10582 raft_consensus.cc:729] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091aa27fc3b245f8949b5d1a7a002120, State: Initialized, Role: FOLLOWER
I20260809 01:52:46.460825 10579 consensus_queue.cc:260] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.460995 10582 consensus_queue.cc:260] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.461791 10556 heartbeater.cc:503] Master 127.3.220.62:35069 was elected leader, sending a full tablet report...
I20260809 01:52:46.462103 10579 ts_tablet_manager.cc:1428] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 01:52:46.462198 10582 ts_tablet_manager.cc:1428] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 01:52:46.462427 10165 heartbeater.cc:503] Master 127.3.220.62:35069 was elected leader, sending a full tablet report...
I20260809 01:52:46.463073 10580 raft_consensus.cc:348] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.463320 10580 raft_consensus.cc:374] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:46.463397 10580 raft_consensus.cc:729] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b45d3a12de440d8552941b9eaacd74, State: Initialized, Role: FOLLOWER
I20260809 01:52:46.464313 10580 consensus_queue.cc:260] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.465029 10354 heartbeater.cc:503] Master 127.3.220.62:35069 was elected leader, sending a full tablet report...
I20260809 01:52:46.465342 10580 ts_tablet_manager.cc:1428] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:52:46.540077 10588 raft_consensus.cc:482] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:46.540288 10588 raft_consensus.cc:504] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.541381 10588 leader_election.cc:288] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08b45d3a12de440d8552941b9eaacd74 (127.3.220.2:44977), 30251326617c485a99204ab03bae5fe3 (127.3.220.1:36153)
I20260809 01:52:46.546793 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.541792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:39873 (local address 127.3.220.1:36153)
0809 01:52:46.541999 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.542005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.542226 (+ 221us) server_negotiation.cc:407] Connection header received
0809 01:52:46.542320 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.542323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.542380 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.542454 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:46.542820 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.543625 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.545421 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.546070 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.546185 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.546253 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.546580 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.546654 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 01:52:46.546793 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.541792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:56051 (local address 127.3.220.2:44977)
0809 01:52:46.541954 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.541960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.542057 (+ 97us) server_negotiation.cc:407] Connection header received
0809 01:52:46.542229 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.542232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.542312 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.542401 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:46.542822 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.543626 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.545421 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.546070 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.546185 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.546253 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.546580 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.546654 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 01:52:46.546828 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.541706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:44977 (local address 127.3.220.3:56051)
0809 01:52:46.541954 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.541971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:46.542087 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:46.542446 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:46.542450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:46.542472 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:46.542694 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.542701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.543783 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.543787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.545291 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.545297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.546232 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.546236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.546429 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.546471 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 01:52:46.546549 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 01:52:46.546654 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":206,"spinlock_wait_cycles":4864}
I20260809 01:52:46.546836 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.541706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:36153 (local address 127.3.220.3:39873)
0809 01:52:46.542119 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.542132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:46.542240 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:46.542481 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:46.542483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:46.542514 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:46.542685 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.542693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.543783 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.543787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.545283 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:46.545296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.546232 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.546236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:46.546420 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.546468 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 01:52:46.546549 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 01:52:46.546654 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":263,"spinlock_wait_cycles":1792,"thread_start_us":159,"threads_started":1}
I20260809 01:52:46.547530 10291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67b5744b0c410b8706cd8ee74c60d1" candidate_uuid: "091aa27fc3b245f8949b5d1a7a002120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b45d3a12de440d8552941b9eaacd74" is_pre_election: true
I20260809 01:52:46.547533 10095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67b5744b0c410b8706cd8ee74c60d1" candidate_uuid: "091aa27fc3b245f8949b5d1a7a002120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30251326617c485a99204ab03bae5fe3" is_pre_election: true
I20260809 01:52:46.547863 10291 raft_consensus.cc:2454] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 091aa27fc3b245f8949b5d1a7a002120 in term 0.
I20260809 01:52:46.547863 10095 raft_consensus.cc:2454] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 091aa27fc3b245f8949b5d1a7a002120 in term 0.
I20260809 01:52:46.548386 10394 leader_election.cc:302] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b45d3a12de440d8552941b9eaacd74, 091aa27fc3b245f8949b5d1a7a002120; no voters:
I20260809 01:52:46.548688 10588 raft_consensus.cc:2781] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:46.548779 10588 raft_consensus.cc:482] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:46.548821 10588 raft_consensus.cc:3037] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:46.549993 10588 raft_consensus.cc:504] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.550347 10588 leader_election.cc:288] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [CANDIDATE]: Term 1 election: Requested vote from peers 08b45d3a12de440d8552941b9eaacd74 (127.3.220.2:44977), 30251326617c485a99204ab03bae5fe3 (127.3.220.1:36153)
I20260809 01:52:46.550729 10291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67b5744b0c410b8706cd8ee74c60d1" candidate_uuid: "091aa27fc3b245f8949b5d1a7a002120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b45d3a12de440d8552941b9eaacd74"
I20260809 01:52:46.550750 10095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc67b5744b0c410b8706cd8ee74c60d1" candidate_uuid: "091aa27fc3b245f8949b5d1a7a002120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30251326617c485a99204ab03bae5fe3"
I20260809 01:52:46.550858 10095 raft_consensus.cc:3037] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:46.550858 10291 raft_consensus.cc:3037] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:46.552186 10095 raft_consensus.cc:2454] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091aa27fc3b245f8949b5d1a7a002120 in term 1.
I20260809 01:52:46.552179 10291 raft_consensus.cc:2454] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091aa27fc3b245f8949b5d1a7a002120 in term 1.
I20260809 01:52:46.552572 10389 leader_election.cc:302] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091aa27fc3b245f8949b5d1a7a002120, 30251326617c485a99204ab03bae5fe3; no voters:
I20260809 01:52:46.552742 10588 raft_consensus.cc:2781] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:46.552990 10588 raft_consensus.cc:686] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 1 LEADER]: Becoming Leader. State: Replica: 091aa27fc3b245f8949b5d1a7a002120, State: Running, Role: LEADER
I20260809 01:52:46.553318 10588 consensus_queue.cc:237] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } }
I20260809 01:52:46.556483 9890 catalog_manager.cc:5495] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 reported cstate change: term changed from 0 to 1, leader changed from <none> to 091aa27fc3b245f8949b5d1a7a002120 (127.3.220.3). New cstate: current_term: 1 leader_uuid: "091aa27fc3b245f8949b5d1a7a002120" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "091aa27fc3b245f8949b5d1a7a002120" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20260809 01:52:46.590019 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.586251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.3.220.3:37921)
0809 01:52:46.586405 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.586409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.586424 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:52:46.586590 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.586593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.586660 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.586787 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:46.587055 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.587574 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.588460 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:46.588694 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:46.588816 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:46.589143 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:46.589633 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:46.589775 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.589813 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.589869 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 01:52:46.596444 10095 raft_consensus.cc:1261] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 1 FOLLOWER]: Refusing update from remote peer 091aa27fc3b245f8949b5d1a7a002120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:46.596444 10291 raft_consensus.cc:1261] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 1 FOLLOWER]: Refusing update from remote peer 091aa27fc3b245f8949b5d1a7a002120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:46.597285 10588 consensus_queue.cc:1035] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30251326617c485a99204ab03bae5fe3" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:46.597514 10593 consensus_queue.cc:1035] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b45d3a12de440d8552941b9eaacd74" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:46.605476 10598 mvcc.cc:205] Tried to move back new op lower bound from 7316440541571031040 to 7316440541405327360. Current Snapshot: MvccSnapshot[applied={T|T < 7316440541571031040}]
I20260809 01:52:46.606842 10597 mvcc.cc:205] Tried to move back new op lower bound from 7316440541571031040 to 7316440541405327360. Current Snapshot: MvccSnapshot[applied={T|T < 7316440541571031040 or (T in {7316440541571031040})}]
I20260809 01:52:46.608312 10599 mvcc.cc:205] Tried to move back new op lower bound from 7316440541571031040 to 7316440541405327360. Current Snapshot: MvccSnapshot[applied={T|T < 7316440541571031040}]
I20260809 01:52:46.613705 9889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:46.613888 9889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:46.616261 9889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc2847544c164608818957cc383322e6: Sending DeleteTablet for 3 replicas of tablet dc67b5744b0c410b8706cd8ee74c60d1
I20260809 01:52:46.617050 10260 tablet_service.cc:1514] Processing DeleteTablet for tablet dc67b5744b0c410b8706cd8ee74c60d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:46 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:60560
I20260809 01:52:46.617100 10453 tablet_service.cc:1514] Processing DeleteTablet for tablet dc67b5744b0c410b8706cd8ee74c60d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:46 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:35306
I20260809 01:52:46.617461 10068 tablet_service.cc:1514] Processing DeleteTablet for tablet dc67b5744b0c410b8706cd8ee74c60d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:46 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:41404
I20260809 01:52:46.617602 10609 tablet_replica.cc:330] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74: stopping tablet replica
I20260809 01:52:46.617981 10609 raft_consensus.cc:2229] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:46.618040 10613 tablet_replica.cc:330] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3: stopping tablet replica
I20260809 01:52:46.618345 10613 raft_consensus.cc:2229] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:46.618428 10609 raft_consensus.cc:2258] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:46.618714 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 9981
I20260809 01:52:46.618788 10607 tablet_replica.cc:330] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: stopping tablet replica
I20260809 01:52:46.618778 10613 raft_consensus.cc:2258] T dc67b5744b0c410b8706cd8ee74c60d1 P 30251326617c485a99204ab03bae5fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:46.619055 10607 raft_consensus.cc:2229] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:52:46.619472 10607 raft_consensus.cc:2258] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:46.622445 10607 ts_tablet_manager.cc:1905] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:46.624614 10607 ts_tablet_manager.cc:1918] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:46.624686 10607 log.cc:1196] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/dc67b5744b0c410b8706cd8ee74c60d1
I20260809 01:52:46.624929 10607 ts_tablet_manager.cc:1939] T dc67b5744b0c410b8706cd8ee74c60d1 P 091aa27fc3b245f8949b5d1a7a002120: Deleting consensus metadata
I20260809 01:52:46.630553 9872 catalog_manager.cc:4841] TS 091aa27fc3b245f8949b5d1a7a002120 (127.3.220.3:37921): tablet dc67b5744b0c410b8706cd8ee74c60d1 (table test-table [id=916f56c484564302b63e8c68d5ad9bbd]) successfully deleted
I20260809 01:52:46.635694 10609 ts_tablet_manager.cc:1905] T dc67b5744b0c410b8706cd8ee74c60d1 P 08b45d3a12de440d8552941b9eaacd74: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 01:52:46.636662 9869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:36153 (error 108)
I20260809 01:52:46.637775 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.637437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:36153 (local address 127.0.0.1:41414)
0809 01:52:46.637594 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.637674 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:36153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 01:52:46.637930 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 10173
W20260809 01:52:46.645071 9869 catalog_manager.cc:4584] TS 30251326617c485a99204ab03bae5fe3 (127.3.220.1:36153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc67b5744b0c410b8706cd8ee74c60d1: Network error: Client connection negotiation failed: client connection to 127.3.220.1:36153: connect: Connection refused (error 111)
I20260809 01:52:46.651876 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 10362
I20260809 01:52:46.652112 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.651689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:44977 (local address 127.0.0.1:60572)
0809 01:52:46.651925 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:46.651992 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.2:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260809 01:52:46.669881 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 9848
2026-08-09T01:52:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2654 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T01:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:46Z Disabled control of system clock
I20260809 01:52:46.714424 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39745
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:39745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 01:52:46.827888 10621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:46.828152 10621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:46.828190 10621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:46.832077 10621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:46.832129 10621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:46.832144 10621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:46.832161 10621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:46.832176 10621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:46.836753 10621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:39745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:46.837798 10621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:46.839166 10621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:46.847888 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:46.847934 10630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:46.847955 10621 server_base.cc:1034] running on GCE node
W20260809 01:52:46.847940 10628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:46.848819 10621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:46.850060 10621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:46.851279 10621 hybrid_clock.cc:648] HybridClock initialized: now 1786240366851247 us; error 47 us; skew 500 ppm
I20260809 01:52:46.853183 10621 webserver.cc:460] Webserver started at http://127.3.220.62:37067/ using document root <none> and password file <none>
I20260809 01:52:46.853869 10621 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:46.853976 10621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:46.854236 10621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:46.856328 10621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "5f107b11c3e449918c082e85194783ed"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "6f30b750516431c3c408bf4497b9544e"
server_key_iv: "afe1c1b3b6bcb5de49143906119b712e"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.856912 10621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "5f107b11c3e449918c082e85194783ed"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "6f30b750516431c3c408bf4497b9544e"
server_key_iv: "afe1c1b3b6bcb5de49143906119b712e"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.860737 10621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260809 01:52:46.863361 10640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:46.864683 10621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:46.864848 10621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "5f107b11c3e449918c082e85194783ed"
format_stamp: "Formatted at 2026-08-09 01:52:46 on dist-test-slave-gkw7"
server_key: "6f30b750516431c3c408bf4497b9544e"
server_key_iv: "afe1c1b3b6bcb5de49143906119b712e"
server_key_version: "encryptionkey@0"
I20260809 01:52:46.864966 10621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:46.869621 10621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:46.870226 10621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:46.870411 10621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:46.877295 10621 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:39745
I20260809 01:52:46.877322 10715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:39745 every 8 connection(s)
I20260809 01:52:46.878283 10621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:46.880445 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 10621
I20260809 01:52:46.880558 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:46.880864 3952 external_mini_cluster.cc:1428] Setting key 451a9d7a7b4e1be9ee22956ebd937e64
I20260809 01:52:46.882448 10716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:46.886662 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:46.882356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.3.220.62:39745)
0809 01:52:46.883025 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:46.883042 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:46.883079 (+ 37us) server_negotiation.cc:407] Connection header received
0809 01:52:46.883751 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:46.883768 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:46.884109 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:46.884389 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:46.885268 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:46.885272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:46.885283 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:46.885293 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:46.885349 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:46.885365 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:46.885476 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:46.885657 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 01:52:46.885918 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":167,"threads_started":1}
I20260809 01:52:46.889173 10716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed: Bootstrap starting.
I20260809 01:52:46.891403 10716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:46.892138 10716 log.cc:824] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:46.894120 10716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed: No bootstrap required, opened a new log
I20260809 01:52:46.897775 10716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f107b11c3e449918c082e85194783ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39745 } }
I20260809 01:52:46.898033 10716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:46.898126 10716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f107b11c3e449918c082e85194783ed, State: Initialized, Role: FOLLOWER
I20260809 01:52:46.898562 10716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f107b11c3e449918c082e85194783ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39745 } }
I20260809 01:52:46.898736 10716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:46.898810 10716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:46.898958 10716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:46.899885 10716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f107b11c3e449918c082e85194783ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39745 } }
I20260809 01:52:46.900236 10716 leader_election.cc:302] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f107b11c3e449918c082e85194783ed; no voters:
I20260809 01:52:46.900599 10716 leader_election.cc:288] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:46.900722 10721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:46.900956 10721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [term 1 LEADER]: Becoming Leader. State: Replica: 5f107b11c3e449918c082e85194783ed, State: Running, Role: LEADER
I20260809 01:52:46.901355 10721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f107b11c3e449918c082e85194783ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39745 } }
I20260809 01:52:46.901933 10716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:46.903208 10722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f107b11c3e449918c082e85194783ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f107b11c3e449918c082e85194783ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39745 } } }
I20260809 01:52:46.903208 10723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f107b11c3e449918c082e85194783ed. Latest consensus state: current_term: 1 leader_uuid: "5f107b11c3e449918c082e85194783ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f107b11c3e449918c082e85194783ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39745 } } }
I20260809 01:52:46.903523 10723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:46.903523 10722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed [sys.catalog]: This master's current role is: LEADER
W20260809 01:52:46.906929 10742 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:52:46.907014 10742 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:52:46.907091 10735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:46.907996 10735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:46.913136 10735 catalog_manager.cc:1347] Generated new cluster ID: ad527d834185453bb58edef8f273ef8f
I20260809 01:52:46.913199 10735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:46.924602 10735 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:46.924933 10735 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:46.933094 10735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed: Generated new TSK 0
I20260809 01:52:46.933885 10735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:52:46.941974 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39745
--builtin_ntp_servers=127.3.220.20:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 01:52:47.058382 10746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:47.058647 10746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:47.058729 10746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:47.062816 10746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:47.062882 10746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:47.062969 10746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:47.067907 10746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:47.069219 10746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:47.070753 10746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:47.080868 10757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:47.081041 10746 server_base.cc:1034] running on GCE node
W20260809 01:52:47.080888 10755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:47.080924 10753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:47.081799 10746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:47.082481 10746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:47.083666 10746 hybrid_clock.cc:648] HybridClock initialized: now 1786240367083656 us; error 36 us; skew 500 ppm
I20260809 01:52:47.085801 10746 webserver.cc:460] Webserver started at http://127.3.220.1:46489/ using document root <none> and password file <none>
I20260809 01:52:47.086401 10746 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:47.086468 10746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:47.086658 10746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:47.088995 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "b8e2c97248a846088f67b6be0a909ffa"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "166a14ef4253cad770703e20e89c3b24"
server_key_iv: "9c8cdb84ecc10ed1e9729d64f72450e7"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.089646 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "b8e2c97248a846088f67b6be0a909ffa"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "166a14ef4253cad770703e20e89c3b24"
server_key_iv: "9c8cdb84ecc10ed1e9729d64f72450e7"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.093722 10746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:52:47.096292 10765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.097674 10746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:47.097810 10746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "b8e2c97248a846088f67b6be0a909ffa"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "166a14ef4253cad770703e20e89c3b24"
server_key_iv: "9c8cdb84ecc10ed1e9729d64f72450e7"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.097923 10746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:47.102767 10746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:47.103456 10746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:47.103645 10746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:47.104272 10746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:47.105269 10746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:47.105317 10746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.105386 10746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:47.105415 10746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.114485 10746 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:46511
I20260809 01:52:47.114511 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:46511 every 8 connection(s)
I20260809 01:52:47.115465 10746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:47.119738 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 10746
I20260809 01:52:47.119916 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:47.120229 3952 external_mini_cluster.cc:1428] Setting key 3c403ec56879e0fd5a5a140ac2b6110e
I20260809 01:52:47.122427 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.117389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:52377 (local address 127.3.220.62:39745)
0809 01:52:47.117594 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.117599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.118697 (+ 1098us) server_negotiation.cc:407] Connection header received
0809 01:52:47.120387 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.120390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.120430 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.120510 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.121919 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.121922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.121925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.121938 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.121962 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.121971 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.122063 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.122247 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.122291 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 01:52:47.122542 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39745
--builtin_ntp_servers=127.3.220.20:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 01:52:47.123569 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.117852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39745 (local address 127.3.220.1:52377)
0809 01:52:47.118535 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.118579 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:47.120120 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:47.120736 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:47.120743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:47.120865 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:47.121578 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:52:47.121582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:47.121645 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:52:47.121744 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:47.122073 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:47.122084 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:47.122090 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 01:52:47.122286 (+ 196us) client_negotiation.cc:240] Negotiation successful
0809 01:52:47.122546 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":126,"threads_started":1}
I20260809 01:52:47.124914 10928 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39745
I20260809 01:52:47.125175 10928 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:47.125715 10928 heartbeater.cc:511] Master 127.3.220.62:39745 requested a full tablet report, sending...
I20260809 01:52:47.127344 10666 ts_manager.cc:194] Registered new tserver with Master: b8e2c97248a846088f67b6be0a909ffa (127.3.220.1:46511)
W20260809 01:52:47.238747 10932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:47.239007 10932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:47.239048 10932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:47.243126 10932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:47.243182 10932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:47.243237 10932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:47.248103 10932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:47.249121 10932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:47.250623 10932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:47.260998 10942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:47.260977 10937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:47.260979 10940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:47.261025 10932 server_base.cc:1034] running on GCE node
I20260809 01:52:47.261749 10932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:47.262351 10932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:47.263567 10932 hybrid_clock.cc:648] HybridClock initialized: now 1786240367263552 us; error 25 us; skew 500 ppm
I20260809 01:52:47.265444 10932 webserver.cc:460] Webserver started at http://127.3.220.2:42491/ using document root <none> and password file <none>
I20260809 01:52:47.266059 10932 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:47.266124 10932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:47.266357 10932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:47.268429 10932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "e17fc858b8894ff994da280c08b3edc3"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "7605c49cfa01209b0040b0c245bddb2d"
server_key_iv: "7336af91740d33238e596029f9851fe9"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.269011 10932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "e17fc858b8894ff994da280c08b3edc3"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "7605c49cfa01209b0040b0c245bddb2d"
server_key_iv: "7336af91740d33238e596029f9851fe9"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.272961 10932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:52:47.275557 10951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.277284 10932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:47.277406 10932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "e17fc858b8894ff994da280c08b3edc3"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "7605c49cfa01209b0040b0c245bddb2d"
server_key_iv: "7336af91740d33238e596029f9851fe9"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.277518 10932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:47.282909 10932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:47.283507 10932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:47.283702 10932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:47.284255 10932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:47.285164 10932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:47.285210 10932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.285274 10932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:47.285311 10932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.294363 10932 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:41931
I20260809 01:52:47.294409 11100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:41931 every 8 connection(s)
I20260809 01:52:47.295293 10932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:47.300560 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 10932
I20260809 01:52:47.300702 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:47.301026 3952 external_mini_cluster.cc:1428] Setting key 5c2feeb6d02b0ab12a6a9ae86f97f107
I20260809 01:52:47.301808 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.297119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:36579 (local address 127.3.220.62:39745)
0809 01:52:47.297328 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.297333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.298468 (+ 1135us) server_negotiation.cc:407] Connection header received
0809 01:52:47.299807 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.299811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.299848 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.299924 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.301243 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.301247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.301250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.301265 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.301285 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.301296 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.301400 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.301604 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.301645 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 01:52:47.302939 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.297628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39745 (local address 127.3.220.2:36579)
0809 01:52:47.298311 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.298354 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:47.299580 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:47.300079 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:47.300085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:47.300203 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:47.300883 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:52:47.300887 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:47.300942 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:52:47.301020 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:47.301389 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:47.301396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:47.301400 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:52:47.301602 (+ 202us) client_negotiation.cc:240] Negotiation successful
0809 01:52:47.301932 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
I20260809 01:52:47.303459 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39745
--builtin_ntp_servers=127.3.220.20:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 01:52:47.304119 11101 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39745
I20260809 01:52:47.304319 11101 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:47.304663 11101 heartbeater.cc:511] Master 127.3.220.62:39745 requested a full tablet report, sending...
I20260809 01:52:47.305692 10666 ts_manager.cc:194] Registered new tserver with Master: e17fc858b8894ff994da280c08b3edc3 (127.3.220.2:41931)
W20260809 01:52:47.418391 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:47.418641 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:47.418679 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:47.422600 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:47.422648 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:47.422701 11104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:47.427342 11104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:47.428387 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:47.429747 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:47.439903 11109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:47.439913 11113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:47.439991 11111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:47.440272 11104 server_base.cc:1034] running on GCE node
I20260809 01:52:47.440663 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:47.441285 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:47.442444 11104 hybrid_clock.cc:648] HybridClock initialized: now 1786240367442415 us; error 42 us; skew 500 ppm
I20260809 01:52:47.444226 11104 webserver.cc:460] Webserver started at http://127.3.220.3:43961/ using document root <none> and password file <none>
I20260809 01:52:47.444804 11104 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:47.444866 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:47.445078 11104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:47.447178 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "5092e001d2464789982f2309ae6c591d"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "aba531bacf5904ba17912200db76da86"
server_key_iv: "01301628685ac76a06a57a4bad61eace"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.447751 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "5092e001d2464789982f2309ae6c591d"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "aba531bacf5904ba17912200db76da86"
server_key_iv: "01301628685ac76a06a57a4bad61eace"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.451421 11104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:52:47.453871 11122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.455053 11104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:52:47.455145 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "5092e001d2464789982f2309ae6c591d"
format_stamp: "Formatted at 2026-08-09 01:52:47 on dist-test-slave-gkw7"
server_key: "aba531bacf5904ba17912200db76da86"
server_key_iv: "01301628685ac76a06a57a4bad61eace"
server_key_version: "encryptionkey@0"
I20260809 01:52:47.455283 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:47.459797 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:47.460397 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:47.460575 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:47.461105 11104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:47.462054 11104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:47.462126 11104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.462200 11104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:47.462252 11104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:47.471580 11104 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:34059
I20260809 01:52:47.471617 11297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:34059 every 8 connection(s)
I20260809 01:52:47.472470 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:47.478684 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.474318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:60213 (local address 127.3.220.62:39745)
0809 01:52:47.474457 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.474460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.475515 (+ 1055us) server_negotiation.cc:407] Connection header received
0809 01:52:47.476827 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.476830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.476875 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.476962 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.478187 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.478190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.478193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.478206 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.478230 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.478242 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.478342 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.478499 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.478547 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:52:47.479557 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.474708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39745 (local address 127.3.220.3:60213)
0809 01:52:47.475375 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.475413 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:47.476603 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:47.477139 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:47.477145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:47.477243 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:47.477920 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:52:47.477921 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:47.477960 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:52:47.478026 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:47.478327 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:47.478335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:47.478340 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 01:52:47.478543 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 01:52:47.478792 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
I20260809 01:52:47.480744 11298 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39745
I20260809 01:52:47.481001 11298 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:47.481499 11298 heartbeater.cc:511] Master 127.3.220.62:39745 requested a full tablet report, sending...
I20260809 01:52:47.481674 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 11104
I20260809 01:52:47.481812 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:47.482093 3952 external_mini_cluster.cc:1428] Setting key 818f1b90e5732e903dbb082af15cf0ac
I20260809 01:52:47.482765 10666 ts_manager.cc:194] Registered new tserver with Master: 5092e001d2464789982f2309ae6c591d (127.3.220.3:34059)
I20260809 01:52:47.484347 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:52:47.487797 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.486258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.3.220.62:39745)
0809 01:52:47.486381 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.486386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.486554 (+ 168us) server_negotiation.cc:407] Connection header received
0809 01:52:47.486649 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.486652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.486684 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.486752 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.487374 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.487377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.487379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.487390 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.487408 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.487415 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.487485 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.487576 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.487606 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 01:52:47.490715 10666 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:52:47.494033 10666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:47.505976 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.502582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:34059 (local address 127.0.0.1:34908)
0809 01:52:47.502901 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.502944 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:47.503079 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:47.504213 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:47.504220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:47.504303 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:47.504882 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:52:47.504884 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:47.504904 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:52:47.504968 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:47.505609 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:47.505621 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:47.505629 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 01:52:47.505766 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 01:52:47.505843 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260809 01:52:47.506079 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.502767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.3.220.3:34059)
0809 01:52:47.503149 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.503162 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.503189 (+ 27us) server_negotiation.cc:407] Connection header received
0809 01:52:47.503246 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.503255 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.503482 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.503649 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.505346 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.505354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.505370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.505387 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.505438 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.505476 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.505615 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.505815 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.505899 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":171,"threads_started":1}
I20260809 01:52:47.506376 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.503717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:41931 (local address 127.0.0.1:44974)
0809 01:52:47.504331 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.504344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:47.504421 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:47.505104 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:47.505106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:47.505118 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:47.505694 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:52:47.505695 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:47.505704 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:52:47.505746 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:47.506147 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:47.506153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:47.506158 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 01:52:47.506246 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 01:52:47.506277 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":97,"threads_started":1}
I20260809 01:52:47.506752 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.504155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.3.220.2:41931)
0809 01:52:47.504669 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.504687 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.504711 (+ 24us) server_negotiation.cc:407] Connection header received
0809 01:52:47.504764 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.504770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.504941 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.505120 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.505897 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.505902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.505912 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.505924 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.505965 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.505998 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.506097 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.506486 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.506570 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":143,"threads_started":1}
I20260809 01:52:47.508395 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.503985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46511 (local address 127.0.0.1:37324)
0809 01:52:47.504619 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.504631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:47.504694 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:47.506898 (+ 2204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:47.506901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:47.506917 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:47.507354 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:52:47.507357 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:47.507368 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:52:47.507398 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:47.508143 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:47.508146 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:47.508149 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 01:52:47.508248 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 01:52:47.508282 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":91,"threads_started":1}
I20260809 01:52:47.508512 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.504148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.3.220.1:46511)
0809 01:52:47.505783 (+ 1635us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.505797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.505820 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:52:47.505871 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.505876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.506094 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.506333 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.507953 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.507959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.507968 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.507979 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.508017 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.508039 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.508146 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.508278 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.508345 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1519,"thread_start_us":107,"threads_started":1}
I20260809 01:52:47.509723 11196 tablet_service.cc:1467] Processing CreateTablet for tablet b2574ddd9a87407f97f609b50ac3464a (DEFAULT_TABLE table=test-table [id=0acd2dd3c2064ce2ba2fc567f6252bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:47.509769 11013 tablet_service.cc:1467] Processing CreateTablet for tablet b2574ddd9a87407f97f609b50ac3464a (DEFAULT_TABLE table=test-table [id=0acd2dd3c2064ce2ba2fc567f6252bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:47.510840 11013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2574ddd9a87407f97f609b50ac3464a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:47.510843 11196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2574ddd9a87407f97f609b50ac3464a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:47.511184 10826 tablet_service.cc:1467] Processing CreateTablet for tablet b2574ddd9a87407f97f609b50ac3464a (DEFAULT_TABLE table=test-table [id=0acd2dd3c2064ce2ba2fc567f6252bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:47.512099 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2574ddd9a87407f97f609b50ac3464a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:47.517069 11322 tablet_bootstrap.cc:492] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Bootstrap starting.
I20260809 01:52:47.519048 11325 tablet_bootstrap.cc:492] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa: Bootstrap starting.
I20260809 01:52:47.519351 11322 tablet_bootstrap.cc:654] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:47.519920 11323 tablet_bootstrap.cc:492] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Bootstrap starting.
I20260809 01:52:47.520169 11322 log.cc:824] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:47.521194 11325 tablet_bootstrap.cc:654] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:47.522025 11322 tablet_bootstrap.cc:492] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: No bootstrap required, opened a new log
I20260809 01:52:47.522032 11323 tablet_bootstrap.cc:654] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:47.522066 11325 log.cc:824] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:47.522241 11322 ts_tablet_manager.cc:1397] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:52:47.522774 11323 log.cc:824] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:47.524003 11325 tablet_bootstrap.cc:492] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa: No bootstrap required, opened a new log
I20260809 01:52:47.524185 11325 ts_tablet_manager.cc:1397] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:52:47.524659 11323 tablet_bootstrap.cc:492] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: No bootstrap required, opened a new log
I20260809 01:52:47.524868 11323 ts_tablet_manager.cc:1397] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:52:47.526326 11322 raft_consensus.cc:348] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.526551 11322 raft_consensus.cc:374] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:47.526628 11322 raft_consensus.cc:729] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5092e001d2464789982f2309ae6c591d, State: Initialized, Role: FOLLOWER
I20260809 01:52:47.527063 11322 consensus_queue.cc:260] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.527875 11322 ts_tablet_manager.cc:1428] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:52:47.528227 11298 heartbeater.cc:503] Master 127.3.220.62:39745 was elected leader, sending a full tablet report...
I20260809 01:52:47.528602 11325 raft_consensus.cc:348] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.528851 11325 raft_consensus.cc:374] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:47.528896 11325 raft_consensus.cc:729] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e2c97248a846088f67b6be0a909ffa, State: Initialized, Role: FOLLOWER
I20260809 01:52:47.529311 11325 consensus_queue.cc:260] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.529255 11323 raft_consensus.cc:348] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.529464 11323 raft_consensus.cc:374] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:47.529588 11323 raft_consensus.cc:729] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17fc858b8894ff994da280c08b3edc3, State: Initialized, Role: FOLLOWER
I20260809 01:52:47.529935 10928 heartbeater.cc:503] Master 127.3.220.62:39745 was elected leader, sending a full tablet report...
I20260809 01:52:47.530056 11323 consensus_queue.cc:260] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.530310 11325 ts_tablet_manager.cc:1428] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:52:47.530800 11101 heartbeater.cc:503] Master 127.3.220.62:39745 was elected leader, sending a full tablet report...
I20260809 01:52:47.530941 11323 ts_tablet_manager.cc:1428] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260809 01:52:47.546308 11102 tablet.cc:2369] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 01:52:47.616693 10929 tablet.cc:2369] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 01:52:47.723487 11299 tablet.cc:2369] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:47.735455 11334 raft_consensus.cc:482] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:47.735757 11334 raft_consensus.cc:504] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.737084 11334 leader_election.cc:288] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5092e001d2464789982f2309ae6c591d (127.3.220.3:34059), e17fc858b8894ff994da280c08b3edc3 (127.3.220.2:41931)
I20260809 01:52:47.739341 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.737402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:52499 (local address 127.3.220.3:34059)
0809 01:52:47.737550 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.737556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.737747 (+ 191us) server_negotiation.cc:407] Connection header received
0809 01:52:47.737870 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.737873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.737934 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.738021 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.738898 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.738901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.738904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.738917 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.738938 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.738948 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.739041 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.739165 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.739211 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 01:52:47.739341 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.737657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:49119 (local address 127.3.220.2:41931)
0809 01:52:47.737848 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.737852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.737942 (+ 90us) server_negotiation.cc:407] Connection header received
0809 01:52:47.737998 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.738004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.738043 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.738124 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.738898 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.738901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.738904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.738917 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.738938 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.738948 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.739041 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.739163 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.739229 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 01:52:47.739388 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.737275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:34059 (local address 127.3.220.1:52499)
0809 01:52:47.737640 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.737657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:47.737760 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:47.738069 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:47.738071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:47.738083 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:47.738634 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:52:47.738675 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:47.738693 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:52:47.738727 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:47.739051 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:47.739054 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:47.739058 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:52:47.739150 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 01:52:47.739194 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":91,"threads_started":1}
I20260809 01:52:47.739389 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.737351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:41931 (local address 127.3.220.1:49119)
0809 01:52:47.737853 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.737867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:47.737940 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:47.738160 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:47.738164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:47.738180 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:47.738725 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:52:47.738726 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:47.738734 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:52:47.738753 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:47.739051 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:47.739054 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:47.739058 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:52:47.739177 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 01:52:47.739202 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":96,"threads_started":1}
I20260809 01:52:47.740095 11231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2574ddd9a87407f97f609b50ac3464a" candidate_uuid: "b8e2c97248a846088f67b6be0a909ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5092e001d2464789982f2309ae6c591d" is_pre_election: true
I20260809 01:52:47.740407 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2574ddd9a87407f97f609b50ac3464a" candidate_uuid: "b8e2c97248a846088f67b6be0a909ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17fc858b8894ff994da280c08b3edc3" is_pre_election: true
I20260809 01:52:47.740446 11231 raft_consensus.cc:2454] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e2c97248a846088f67b6be0a909ffa in term 0.
I20260809 01:52:47.740690 11042 raft_consensus.cc:2454] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e2c97248a846088f67b6be0a909ffa in term 0.
I20260809 01:52:47.740967 10768 leader_election.cc:302] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5092e001d2464789982f2309ae6c591d, b8e2c97248a846088f67b6be0a909ffa; no voters:
I20260809 01:52:47.741250 11334 raft_consensus.cc:2781] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:47.741335 11334 raft_consensus.cc:482] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:47.741379 11334 raft_consensus.cc:3037] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:47.742515 11334 raft_consensus.cc:504] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.742854 11334 leader_election.cc:288] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [CANDIDATE]: Term 1 election: Requested vote from peers 5092e001d2464789982f2309ae6c591d (127.3.220.3:34059), e17fc858b8894ff994da280c08b3edc3 (127.3.220.2:41931)
I20260809 01:52:47.743139 11231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2574ddd9a87407f97f609b50ac3464a" candidate_uuid: "b8e2c97248a846088f67b6be0a909ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5092e001d2464789982f2309ae6c591d"
I20260809 01:52:47.743253 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2574ddd9a87407f97f609b50ac3464a" candidate_uuid: "b8e2c97248a846088f67b6be0a909ffa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17fc858b8894ff994da280c08b3edc3"
I20260809 01:52:47.743292 11231 raft_consensus.cc:3037] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:47.743374 11042 raft_consensus.cc:3037] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:47.744619 11231 raft_consensus.cc:2454] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e2c97248a846088f67b6be0a909ffa in term 1.
I20260809 01:52:47.744619 11042 raft_consensus.cc:2454] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e2c97248a846088f67b6be0a909ffa in term 1.
I20260809 01:52:47.744948 10771 leader_election.cc:302] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8e2c97248a846088f67b6be0a909ffa, e17fc858b8894ff994da280c08b3edc3; no voters:
I20260809 01:52:47.745122 11334 raft_consensus.cc:2781] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:47.745388 11334 raft_consensus.cc:686] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [term 1 LEADER]: Becoming Leader. State: Replica: b8e2c97248a846088f67b6be0a909ffa, State: Running, Role: LEADER
I20260809 01:52:47.745735 11334 consensus_queue.cc:237] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } }
I20260809 01:52:47.748832 10663 catalog_manager.cc:5495] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa reported cstate change: term changed from 0 to 1, leader changed from <none> to b8e2c97248a846088f67b6be0a909ffa (127.3.220.1). New cstate: current_term: 1 leader_uuid: "b8e2c97248a846088f67b6be0a909ffa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8e2c97248a846088f67b6be0a909ffa" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46511 } health_report { overall_health: HEALTHY } } }
I20260809 01:52:47.799297 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.797722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.3.220.1:46511)
0809 01:52:47.797861 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:47.797865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:47.797879 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:47.798005 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:47.798008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:47.798067 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:47.798140 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:47.798885 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:47.798889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:47.798890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:47.798901 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:47.798920 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:47.798928 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:47.799001 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:47.799067 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 01:52:47.799100 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 01:52:47.806365 11042 raft_consensus.cc:1261] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 1 FOLLOWER]: Refusing update from remote peer b8e2c97248a846088f67b6be0a909ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:47.806365 11231 raft_consensus.cc:1261] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 1 FOLLOWER]: Refusing update from remote peer b8e2c97248a846088f67b6be0a909ffa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:47.807282 11334 consensus_queue.cc:1035] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e17fc858b8894ff994da280c08b3edc3" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:47.807583 11346 consensus_queue.cc:1035] T b2574ddd9a87407f97f609b50ac3464a P b8e2c97248a846088f67b6be0a909ffa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5092e001d2464789982f2309ae6c591d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:47.816957 11351 mvcc.cc:205] Tried to move back new op lower bound from 7316440546525663232 to 7316440546288992256. Current Snapshot: MvccSnapshot[applied={T|T < 7316440546525663232}]
I20260809 01:52:47.824986 10666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:47.825148 10666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:47.827860 10666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f107b11c3e449918c082e85194783ed: Sending DeleteTablet for 3 replicas of tablet b2574ddd9a87407f97f609b50ac3464a
I20260809 01:52:47.828787 11013 tablet_service.cc:1514] Processing DeleteTablet for tablet b2574ddd9a87407f97f609b50ac3464a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:47 UTC) from {username='slave'} at 127.0.0.1:44974
I20260809 01:52:47.828938 11196 tablet_service.cc:1514] Processing DeleteTablet for tablet b2574ddd9a87407f97f609b50ac3464a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:47 UTC) from {username='slave'} at 127.0.0.1:34908
I20260809 01:52:47.829348 11362 tablet_replica.cc:330] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: stopping tablet replica
I20260809 01:52:47.829366 10826 tablet_service.cc:1514] Processing DeleteTablet for tablet b2574ddd9a87407f97f609b50ac3464a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:47 UTC) from {username='slave'} at 127.0.0.1:37324
I20260809 01:52:47.829813 11362 raft_consensus.cc:2229] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:47.829869 11363 tablet_replica.cc:330] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: stopping tablet replica
I20260809 01:52:47.830261 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 10746
I20260809 01:52:47.837129 11363 raft_consensus.cc:2229] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:47.837719 11363 raft_consensus.cc:2258] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:47.840531 11363 ts_tablet_manager.cc:1905] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:47.844201 11363 ts_tablet_manager.cc:1918] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:47.844329 11363 log.cc:1196] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/b2574ddd9a87407f97f609b50ac3464a
I20260809 01:52:47.844890 11363 ts_tablet_manager.cc:1939] T b2574ddd9a87407f97f609b50ac3464a P 5092e001d2464789982f2309ae6c591d: Deleting consensus metadata
I20260809 01:52:47.845564 11362 raft_consensus.cc:2258] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:47.846707 10646 catalog_manager.cc:4841] TS 5092e001d2464789982f2309ae6c591d (127.3.220.3:34059): tablet b2574ddd9a87407f97f609b50ac3464a (table test-table [id=0acd2dd3c2064ce2ba2fc567f6252bf4]) successfully deleted
I20260809 01:52:47.847434 11362 ts_tablet_manager.cc:1905] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:47.851014 11362 ts_tablet_manager.cc:1918] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:47.851132 11362 log.cc:1196] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/b2574ddd9a87407f97f609b50ac3464a
I20260809 01:52:47.851467 11362 ts_tablet_manager.cc:1939] T b2574ddd9a87407f97f609b50ac3464a P e17fc858b8894ff994da280c08b3edc3: Deleting consensus metadata
I20260809 01:52:47.852910 10647 catalog_manager.cc:4841] TS e17fc858b8894ff994da280c08b3edc3 (127.3.220.2:41931): tablet b2574ddd9a87407f97f609b50ac3464a (table test-table [id=0acd2dd3c2064ce2ba2fc567f6252bf4]) successfully deleted
W20260809 01:52:47.853036 10646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:46511 (error 108)
I20260809 01:52:47.853912 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 10932
I20260809 01:52:47.854305 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:47.853918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46511 (local address 127.0.0.1:37344)
0809 01:52:47.854095 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:47.854194 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:46511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 01:52:47.858215 10646 catalog_manager.cc:4584] TS b8e2c97248a846088f67b6be0a909ffa (127.3.220.1:46511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2574ddd9a87407f97f609b50ac3464a: Network error: Client connection negotiation failed: client connection to 127.3.220.1:46511: connect: Connection refused (error 111)
I20260809 01:52:47.868314 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 11104
I20260809 01:52:47.887526 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 10621
2026-08-09T01:52:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1217 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 01:52:47.921831 3952 mini_oidc.cc:152] Starting JWKS server
I20260809 01:52:47.923655 3952 webserver.cc:460] Webserver started at https://127.0.0.1:42163/ using document root <none> and password file <none>
I20260809 01:52:47.923842 3952 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 01:52:47.924157 3952 webserver.cc:460] Webserver started at http://127.0.0.1:33939/ using document root <none> and password file <none>
2026-08-09T01:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:47Z Disabled control of system clock
I20260809 01:52:47.935545 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:46649
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:46649
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42163/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/testchainca.cert with env {}
W20260809 01:52:48.051728 11372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:48.052030 11372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:48.052073 11372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:48.056306 11372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:48.056388 11372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:48.056404 11372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:48.056421 11372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:48.056437 11372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 01:52:48.056452 11372 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 01:52:48.056511 11372 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42163/jwks.json
I20260809 01:52:48.062579 11372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:46649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:46649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42163/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:48.063786 11372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:48.064878 11372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:48.074023 11386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:48.074046 11383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:48.074010 11380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:48.074342 11372 server_base.cc:1034] running on GCE node
I20260809 01:52:48.075096 11372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:48.076362 11372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:48.077612 11372 hybrid_clock.cc:648] HybridClock initialized: now 1786240368077591 us; error 28 us; skew 500 ppm
I20260809 01:52:48.079773 11372 webserver.cc:460] Webserver started at http://127.3.220.62:35995/ using document root <none> and password file <none>
I20260809 01:52:48.080410 11372 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:48.080513 11372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:48.080765 11372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:48.082935 11372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "cb9e93b0b8614dde84a2c17fcd7c396a"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "d28674b3f9963f77ccbefa69bdfcb163"
server_key_iv: "6bff98fb92cf8493c1560d146aa466f1"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.083525 11372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "cb9e93b0b8614dde84a2c17fcd7c396a"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "d28674b3f9963f77ccbefa69bdfcb163"
server_key_iv: "6bff98fb92cf8493c1560d146aa466f1"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.087654 11372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:52:48.090296 11393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:48.091639 11372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:48.091806 11372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "cb9e93b0b8614dde84a2c17fcd7c396a"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "d28674b3f9963f77ccbefa69bdfcb163"
server_key_iv: "6bff98fb92cf8493c1560d146aa466f1"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.091929 11372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:48.122149 11372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:48.122895 11372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:48.123061 11372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:48.130224 11372 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:46649
I20260809 01:52:48.130234 11460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:46649 every 8 connection(s)
I20260809 01:52:48.131234 11372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:48.132941 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 11372
I20260809 01:52:48.133097 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:48.133388 3952 external_mini_cluster.cc:1428] Setting key f8ac5e99d3bc155de694d04397d69b49
I20260809 01:52:48.134918 11463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:48.141319 11463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a: Bootstrap starting.
I20260809 01:52:48.142902 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:48.134804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.3.220.62:46649)
0809 01:52:48.135451 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:48.135467 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:48.135504 (+ 37us) server_negotiation.cc:407] Connection header received
0809 01:52:48.136144 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:48.136160 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:48.136576 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:48.136864 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:48.137241 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.138289 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.139526 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.139984 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.140109 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.141011 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:48.141016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:48.141029 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:48.141473 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:48.141572 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:48.141605 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:48.141684 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:48.141860 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 01:52:48.142162 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":149,"threads_started":1}
I20260809 01:52:48.143873 11463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:48.144654 11463 log.cc:824] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:48.145886 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:52:48.146953 11463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a: No bootstrap required, opened a new log
I20260809 01:52:48.150619 11463 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9e93b0b8614dde84a2c17fcd7c396a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46649 } }
I20260809 01:52:48.150847 11463 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:48.150938 11463 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9e93b0b8614dde84a2c17fcd7c396a, State: Initialized, Role: FOLLOWER
I20260809 01:52:48.151352 11463 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9e93b0b8614dde84a2c17fcd7c396a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46649 } }
I20260809 01:52:48.151495 11463 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:48.151564 11463 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:48.151686 11463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:48.152639 11463 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9e93b0b8614dde84a2c17fcd7c396a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46649 } }
I20260809 01:52:48.153016 11463 leader_election.cc:302] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb9e93b0b8614dde84a2c17fcd7c396a; no voters:
I20260809 01:52:48.153431 11463 leader_election.cc:288] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:48.153591 11470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:48.153872 11470 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [term 1 LEADER]: Becoming Leader. State: Replica: cb9e93b0b8614dde84a2c17fcd7c396a, State: Running, Role: LEADER
I20260809 01:52:48.154223 11470 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9e93b0b8614dde84a2c17fcd7c396a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46649 } }
I20260809 01:52:48.154781 11463 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:48.155879 11473 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb9e93b0b8614dde84a2c17fcd7c396a. Latest consensus state: current_term: 1 leader_uuid: "cb9e93b0b8614dde84a2c17fcd7c396a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9e93b0b8614dde84a2c17fcd7c396a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46649 } } }
I20260809 01:52:48.156013 11473 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:48.156416 11487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:48.156391 11472 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb9e93b0b8614dde84a2c17fcd7c396a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9e93b0b8614dde84a2c17fcd7c396a" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46649 } } }
I20260809 01:52:48.156634 11472 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:48.159665 11487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:48.165588 11487 catalog_manager.cc:1347] Generated new cluster ID: c9bd543c60924edeaf6512f16aca031a
I20260809 01:52:48.165668 11487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:48.179041 11487 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:48.180294 11487 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:48.188911 11487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb9e93b0b8614dde84a2c17fcd7c396a: Generated new TSK 0
I20260809 01:52:48.189795 11487 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 01:52:48.219180 11518 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 01:52:48.219362 11518 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 01:52:48.220024 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:48.202188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.3.220.62:46649)
0809 01:52:48.202343 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:48.202349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:48.202531 (+ 182us) server_negotiation.cc:407] Connection header received
0809 01:52:48.202736 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:48.202739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:48.202798 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:48.202912 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0809 01:52:48.203144 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.203973 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.204883 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.205231 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.205335 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.205695 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 01:52:48.219583 (+ 13888us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 01:52:48.219876 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59592: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 01:52:48.220095 11466 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59592: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 01:52:48.221383 11517 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:48.202080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:46649 (local address 127.0.0.1:59592)
0809 01:52:48.202445 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:48.202458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:48.202574 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:48.202933 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:48.202935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:48.202942 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0809 01:52:48.203039 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:48.203044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.204091 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.204094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:48.204767 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:48.204775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.205348 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.205351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:48.205413 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.205457 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 01:52:48.219840 (+ 14383us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 01:52:48.220051 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.220.62:46649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.220.62:46649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42163/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 01:52:48.222512 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 11372
2026-08-09T01:52:48Z chronyd exiting
I20260809 01:52:48.267191 3952 test_util.cc:175] -----------------------------------------------
I20260809 01:52:48.267297 3952 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786240338176169-3952-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 01:52:48.268566 3952 mini_oidc.cc:152] Starting JWKS server
I20260809 01:52:48.269623 3952 webserver.cc:460] Webserver started at https://127.0.0.1:39263/ using document root <none> and password file <none>
I20260809 01:52:48.269763 3952 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 01:52:48.270112 3952 webserver.cc:460] Webserver started at http://127.0.0.1:38339/ using document root <none> and password file <none>
2026-08-09T01:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:48Z Disabled control of system clock
I20260809 01:52:48.280665 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44639
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:44639
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39263/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/testchainca.cert with env {}
W20260809 01:52:48.393800 11525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:48.394109 11525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:48.394200 11525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:48.398355 11525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:48.398452 11525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:48.398479 11525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:48.398517 11525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:48.398563 11525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 01:52:48.398599 11525 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 01:52:48.398636 11525 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39263/jwks.json
I20260809 01:52:48.404098 11525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:44639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39263/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:48.405213 11525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:48.406276 11525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:48.415239 11531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:48.415241 11535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:48.415302 11532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:48.415414 11525 server_base.cc:1034] running on GCE node
I20260809 01:52:48.416182 11525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:48.417279 11525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:48.418478 11525 hybrid_clock.cc:648] HybridClock initialized: now 1786240368418459 us; error 40 us; skew 500 ppm
I20260809 01:52:48.420465 11525 webserver.cc:460] Webserver started at http://127.3.220.62:34135/ using document root <none> and password file <none>
I20260809 01:52:48.421058 11525 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:48.421157 11525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:48.421396 11525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:48.423532 11525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "2de8ecdcc35f436b8f40d6ae719f03d8"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "4b789d1b451c10003a65229c153474ea"
server_key_iv: "6b82d8eb0871bdf92f9a740fdc3b8e3d"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.424129 11525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "2de8ecdcc35f436b8f40d6ae719f03d8"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "4b789d1b451c10003a65229c153474ea"
server_key_iv: "6b82d8eb0871bdf92f9a740fdc3b8e3d"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.427793 11525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:52:48.430398 11543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:48.431681 11525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:48.431826 11525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "2de8ecdcc35f436b8f40d6ae719f03d8"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "4b789d1b451c10003a65229c153474ea"
server_key_iv: "6b82d8eb0871bdf92f9a740fdc3b8e3d"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.431941 11525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:48.452565 11525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:48.453351 11525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:48.453632 11525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:48.461136 11525 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:44639
I20260809 01:52:48.461145 11608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:44639 every 8 connection(s)
I20260809 01:52:48.462239 11525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:48.465497 11610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:48.467141 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 11525
I20260809 01:52:48.467278 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:48.467583 3952 external_mini_cluster.cc:1428] Setting key 6152b7316f363a2a104f08b63f1e5ec0
I20260809 01:52:48.472370 11610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8: Bootstrap starting.
I20260809 01:52:48.475220 11610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:48.476082 11610 log.cc:824] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:48.477643 11613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:48.469049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.3.220.62:44639)
0809 01:52:48.469789 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:48.469806 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:48.469844 (+ 38us) server_negotiation.cc:407] Connection header received
0809 01:52:48.470552 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:48.470571 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:48.470986 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:48.471275 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:48.471723 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.472773 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.474064 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.474538 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.474685 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.475638 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:48.475646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:48.475664 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:48.476087 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:48.476182 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:48.476214 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:48.476317 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:48.476497 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 01:52:48.476804 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
I20260809 01:52:48.478286 11610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8: No bootstrap required, opened a new log
I20260809 01:52:48.480564 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:52:48.482388 11610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de8ecdcc35f436b8f40d6ae719f03d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44639 } }
I20260809 01:52:48.482650 11610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:48.482749 11610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de8ecdcc35f436b8f40d6ae719f03d8, State: Initialized, Role: FOLLOWER
I20260809 01:52:48.483182 11610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de8ecdcc35f436b8f40d6ae719f03d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44639 } }
I20260809 01:52:48.483341 11610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:48.483429 11610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:48.483554 11610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:48.484562 11610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de8ecdcc35f436b8f40d6ae719f03d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44639 } }
I20260809 01:52:48.484917 11610 leader_election.cc:302] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2de8ecdcc35f436b8f40d6ae719f03d8; no voters:
I20260809 01:52:48.485294 11610 leader_election.cc:288] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:48.485457 11617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:48.485751 11617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [term 1 LEADER]: Becoming Leader. State: Replica: 2de8ecdcc35f436b8f40d6ae719f03d8, State: Running, Role: LEADER
I20260809 01:52:48.486145 11617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de8ecdcc35f436b8f40d6ae719f03d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44639 } }
I20260809 01:52:48.486676 11610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:48.487991 11618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2de8ecdcc35f436b8f40d6ae719f03d8. Latest consensus state: current_term: 1 leader_uuid: "2de8ecdcc35f436b8f40d6ae719f03d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de8ecdcc35f436b8f40d6ae719f03d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44639 } } }
I20260809 01:52:48.487979 11621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2de8ecdcc35f436b8f40d6ae719f03d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de8ecdcc35f436b8f40d6ae719f03d8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44639 } } }
I20260809 01:52:48.488161 11621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:48.488161 11618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:48.488613 11629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:48.491935 11629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:48.498018 11629 catalog_manager.cc:1347] Generated new cluster ID: 5a34ff1bd34d461a9c12e3f152a761c2
I20260809 01:52:48.498107 11629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:48.524049 11629 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:48.525133 11629 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:48.531342 11629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2de8ecdcc35f436b8f40d6ae719f03d8: Generated new TSK 0
I20260809 01:52:48.532060 11629 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 01:52:48.566416 11655 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 01:52:48.566547 11655 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 01:52:48.567276 11613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:48.544779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.3.220.62:44639)
0809 01:52:48.544922 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:48.544927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:48.545100 (+ 173us) server_negotiation.cc:407] Connection header received
0809 01:52:48.545281 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:48.545285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:48.545343 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:48.545465 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0809 01:52:48.545790 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.546647 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.547587 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.547949 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.548054 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.548427 (+ 373us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 01:52:48.566825 (+ 18398us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 01:52:48.567126 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53798: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 01:52:48.567353 11613 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53798: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 01:52:48.567448 11651 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:48.544701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44639 (local address 127.0.0.1:53798)
0809 01:52:48.545005 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:48.545019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:48.545122 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:48.545495 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:48.545498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:48.545506 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0809 01:52:48.545676 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:48.545682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.546801 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.546804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:48.547465 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:48.547474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.548096 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.548099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:48.548204 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.548231 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 01:52:48.567071 (+ 18840us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 01:52:48.567253 (+ 182us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.220.62:44639: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":121,"threads_started":1}
I20260809 01:52:48.567880 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 11525
2026-08-09T01:52:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (357 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 01:52:48.625504 3952 mini_oidc.cc:152] Starting JWKS server
I20260809 01:52:48.626446 3952 webserver.cc:460] Webserver started at https://127.0.0.1:35027/ using document root <none> and password file <none>
I20260809 01:52:48.626540 3952 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 01:52:48.626742 3952 webserver.cc:460] Webserver started at http://127.0.0.1:42911/ using document root <none> and password file <none>
2026-08-09T01:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:48Z Disabled control of system clock
I20260809 01:52:48.637454 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39101
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:39101
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35027/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 01:52:48.752871 11664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:48.753154 11664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:48.753196 11664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:48.757131 11664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:48.757191 11664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:48.757210 11664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:48.757230 11664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:48.757248 11664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 01:52:48.757267 11664 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 01:52:48.757283 11664 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35027/jwks.json
I20260809 01:52:48.762056 11664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:39101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35027/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:48.763257 11664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:48.764672 11664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:52:48.773978 11664 server_base.cc:1034] running on GCE node
W20260809 01:52:48.773947 11677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:48.774103 11670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:48.773962 11671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:48.774902 11664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:48.775979 11664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:48.777154 11664 hybrid_clock.cc:648] HybridClock initialized: now 1786240368777124 us; error 43 us; skew 500 ppm
I20260809 01:52:48.779167 11664 webserver.cc:460] Webserver started at http://127.3.220.62:39375/ using document root <none> and password file <none>
I20260809 01:52:48.779767 11664 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:48.779827 11664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:48.780078 11664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:48.782595 11664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "4d0f2ec682994e95a93980229dd71122"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "28cf18c88f279fd8c371fa2dcca75b9b"
server_key_iv: "ec57abdf827dae082f9edeb1b28af6ef"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.783267 11664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "4d0f2ec682994e95a93980229dd71122"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "28cf18c88f279fd8c371fa2dcca75b9b"
server_key_iv: "ec57abdf827dae082f9edeb1b28af6ef"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.787591 11664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:52:48.790114 11685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:48.791319 11664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:48.791464 11664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "4d0f2ec682994e95a93980229dd71122"
format_stamp: "Formatted at 2026-08-09 01:52:48 on dist-test-slave-gkw7"
server_key: "28cf18c88f279fd8c371fa2dcca75b9b"
server_key_iv: "ec57abdf827dae082f9edeb1b28af6ef"
server_key_version: "encryptionkey@0"
I20260809 01:52:48.791575 11664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:48.825615 11664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:48.826447 11664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:48.826751 11664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:48.834606 11664 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:39101
I20260809 01:52:48.834635 11762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:39101 every 8 connection(s)
I20260809 01:52:48.835633 11664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:48.839036 11763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:48.844487 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 11664
I20260809 01:52:48.844630 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:48.844928 3952 external_mini_cluster.cc:1428] Setting key 02e532e2a50db5f2e95bd007e68d71b1
I20260809 01:52:48.845585 11763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122: Bootstrap starting.
I20260809 01:52:48.848158 11763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:48.848938 11763 log.cc:824] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:48.850919 11763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122: No bootstrap required, opened a new log
I20260809 01:52:48.854432 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:48.846466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.3.220.62:39101)
0809 01:52:48.847148 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:48.847164 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:48.847201 (+ 37us) server_negotiation.cc:407] Connection header received
0809 01:52:48.847873 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:48.847891 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:48.848327 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:48.848709 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:48.849200 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.850244 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.851146 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.851619 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.851817 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.852675 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:48.852680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:48.852692 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:48.853072 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:48.853130 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:48.853150 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:48.853254 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:48.853386 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 01:52:48.853687 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":151,"threads_started":1}
I20260809 01:52:48.854547 11763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0f2ec682994e95a93980229dd71122" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39101 } }
I20260809 01:52:48.854828 11763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:48.854919 11763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f2ec682994e95a93980229dd71122, State: Initialized, Role: FOLLOWER
I20260809 01:52:48.855429 11763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0f2ec682994e95a93980229dd71122" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39101 } }
I20260809 01:52:48.855589 11763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:48.855654 11763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:48.855751 11763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:48.856925 11763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0f2ec682994e95a93980229dd71122" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39101 } }
I20260809 01:52:48.857301 11763 leader_election.cc:302] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d0f2ec682994e95a93980229dd71122; no voters:
I20260809 01:52:48.857656 11763 leader_election.cc:288] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:48.857733 11771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:48.857752 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:52:48.857934 11771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [term 1 LEADER]: Becoming Leader. State: Replica: 4d0f2ec682994e95a93980229dd71122, State: Running, Role: LEADER
I20260809 01:52:48.858219 11771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0f2ec682994e95a93980229dd71122" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39101 } }
I20260809 01:52:48.858969 11763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:48.859881 11773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d0f2ec682994e95a93980229dd71122. Latest consensus state: current_term: 1 leader_uuid: "4d0f2ec682994e95a93980229dd71122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0f2ec682994e95a93980229dd71122" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39101 } } }
I20260809 01:52:48.860033 11773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:48.860389 11774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d0f2ec682994e95a93980229dd71122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d0f2ec682994e95a93980229dd71122" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39101 } } }
I20260809 01:52:48.860486 11774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122 [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:48.861246 11781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:48.863931 11781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:48.869130 11781 catalog_manager.cc:1347] Generated new cluster ID: 0312469c5b8140f792eddfdd5a00d725
I20260809 01:52:48.869192 11781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:48.904316 11781 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:48.905696 11781 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:48.917953 11781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d0f2ec682994e95a93980229dd71122: Generated new TSK 0
I20260809 01:52:48.918846 11781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:52:48.938222 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:48.932600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.3.220.62:39101)
0809 01:52:48.932748 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:48.932754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:48.932905 (+ 151us) server_negotiation.cc:407] Connection header received
0809 01:52:48.933051 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:48.933055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:48.933122 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:48.933263 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0809 01:52:48.933565 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.934407 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.935396 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.935763 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.935877 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.936180 (+ 303us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 01:52:48.937800 (+ 1620us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 01:52:48.938057 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34510: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 01:52:48.938308 11767 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34510: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 01:52:48.939859 11804 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:48.932439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39101 (local address 127.0.0.1:34510)
0809 01:52:48.932811 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:48.932824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:48.932925 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:48.933292 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:48.933296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:48.933304 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0809 01:52:48.933412 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:48.933417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.934586 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.934589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:48.935262 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:48.935271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:48.935901 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:48.935904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:48.935971 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:48.935994 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 01:52:48.937978 (+ 1984us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 01:52:48.939692 (+ 1714us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.220.62:39101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":1516,"thread_start_us":137,"threads_started":1}
I20260809 01:52:48.940191 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 11664
E20260809 01:52:48.956506 11808 webserver.cc:533] Webserver: sslize failed:
E20260809 01:52:48.956586 11808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-09T01:52:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (362 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T01:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:48Z Disabled control of system clock
I20260809 01:52:48.998692 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:46363
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:46363
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/testchainca.cert with env {}
W20260809 01:52:49.112313 11819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:49.112578 11819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:49.112619 11819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:49.116617 11819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:49.116688 11819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:49.116703 11819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:49.116720 11819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:49.116735 11819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 01:52:49.116752 11819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 01:52:49.116765 11819 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 01:52:49.122331 11819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:46363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:46363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:49.123405 11819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:49.124410 11819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:49.133205 11826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:49.133205 11828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:49.133317 11831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:49.133466 11819 server_base.cc:1034] running on GCE node
I20260809 01:52:49.134114 11819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:49.135269 11819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:49.136473 11819 hybrid_clock.cc:648] HybridClock initialized: now 1786240369136427 us; error 55 us; skew 500 ppm
I20260809 01:52:49.138682 11819 webserver.cc:460] Webserver started at http://127.3.220.62:37031/ using document root <none> and password file <none>
I20260809 01:52:49.139300 11819 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:49.139371 11819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:49.139549 11819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:49.141875 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f"
format_stamp: "Formatted at 2026-08-09 01:52:49 on dist-test-slave-gkw7"
server_key: "8709f41ec2da59cfafd30d5d55f56bc0"
server_key_iv: "b954fd4f792828a0d723593f3c2d1a99"
server_key_version: "encryptionkey@0"
I20260809 01:52:49.142462 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f"
format_stamp: "Formatted at 2026-08-09 01:52:49 on dist-test-slave-gkw7"
server_key: "8709f41ec2da59cfafd30d5d55f56bc0"
server_key_iv: "b954fd4f792828a0d723593f3c2d1a99"
server_key_version: "encryptionkey@0"
I20260809 01:52:49.146502 11819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 01:52:49.149222 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:49.150664 11819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:52:49.150779 11819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f"
format_stamp: "Formatted at 2026-08-09 01:52:49 on dist-test-slave-gkw7"
server_key: "8709f41ec2da59cfafd30d5d55f56bc0"
server_key_iv: "b954fd4f792828a0d723593f3c2d1a99"
server_key_version: "encryptionkey@0"
I20260809 01:52:49.150879 11819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:49.174145 11819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:49.174889 11819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:49.175060 11819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:49.182909 11819 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:46363
I20260809 01:52:49.182935 11915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:46363 every 8 connection(s)
I20260809 01:52:49.183969 11819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:49.185168 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 11819
I20260809 01:52:49.185302 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:49.185647 3952 external_mini_cluster.cc:1428] Setting key ad23de34e8f073e585f927777fdf41ea
I20260809 01:52:49.187932 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:49.193686 11916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f: Bootstrap starting.
I20260809 01:52:49.195726 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:49.187077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.3.220.62:46363)
0809 01:52:49.187791 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:49.187808 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:49.187848 (+ 40us) server_negotiation.cc:407] Connection header received
0809 01:52:49.188596 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:49.188615 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:49.189041 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:49.189413 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:49.189918 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:49.190986 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:49.192049 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:49.192572 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:49.192789 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:49.193854 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:49.193864 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:49.193882 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:52:49.194317 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:49.194375 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:52:49.194398 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:49.194523 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:49.194666 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 01:52:49.194944 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":153,"threads_started":1}
I20260809 01:52:49.196116 11916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:49.196969 11916 log.cc:824] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:49.198897 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:52:49.199460 11916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f: No bootstrap required, opened a new log
I20260809 01:52:49.203776 11916 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46363 } }
I20260809 01:52:49.204103 11916 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:49.204185 11916 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9c3bdf9687411db4bd4d2ec8e0423f, State: Initialized, Role: FOLLOWER
I20260809 01:52:49.204715 11916 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46363 } }
I20260809 01:52:49.204893 11916 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:49.204975 11916 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:49.205087 11916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:49.206251 11916 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46363 } }
I20260809 01:52:49.206636 11916 leader_election.cc:302] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa9c3bdf9687411db4bd4d2ec8e0423f; no voters:
I20260809 01:52:49.206992 11916 leader_election.cc:288] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:49.207122 11923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:49.207366 11923 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [term 1 LEADER]: Becoming Leader. State: Replica: fa9c3bdf9687411db4bd4d2ec8e0423f, State: Running, Role: LEADER
I20260809 01:52:49.207701 11923 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46363 } }
I20260809 01:52:49.208591 11916 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:49.209841 11924 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46363 } } }
I20260809 01:52:49.210011 11924 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:49.210321 11927 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa9c3bdf9687411db4bd4d2ec8e0423f. Latest consensus state: current_term: 1 leader_uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9c3bdf9687411db4bd4d2ec8e0423f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46363 } } }
I20260809 01:52:49.210424 11927 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:49.210546 11938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:49.213449 11938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:49.219014 11938 catalog_manager.cc:1347] Generated new cluster ID: 0e2ec3204a2d47ffa6309862b3424cec
I20260809 01:52:49.219084 11938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:49.250200 11938 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:49.251184 11938 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:49.255789 11938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa9c3bdf9687411db4bd4d2ec8e0423f: Generated new TSK 0
I20260809 01:52:49.256683 11938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:52:49.704555 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:49.268006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.3.220.62:46363)
0809 01:52:49.268145 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:49.268151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:49.268331 (+ 180us) server_negotiation.cc:407] Connection header received
0809 01:52:49.268494 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:49.268496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:49.268544 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:49.268633 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0809 01:52:49.268922 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:49.269450 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:49.270707 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:49.270920 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:49.270997 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:49.271333 (+ 336us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 01:52:49.704066 (+432733us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 01:52:49.704423 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44136: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 01:52:49.704665 11921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44136: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 01:52:49.705091 11964 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:49.267931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:46363 (local address 127.0.0.1:44136)
0809 01:52:49.268235 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:49.268251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:49.268368 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:49.268671 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:49.268675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:49.268684 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 01:52:49.268813 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:49.268820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:49.269599 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:49.269603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:49.270589 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:49.270599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:49.271034 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:49.271038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:49.271137 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:49.271174 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 01:52:49.704374 (+433200us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 01:52:49.704578 (+ 204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.220.62:46363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
I20260809 01:52:49.705966 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 11819
2026-08-09T01:52:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (744 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (129 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:50 dist-test-slave-gkw7 krb5kdc[11983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:50 dist-test-slave-gkw7 krb5kdc[11983](info): set up 2 sockets
Aug 09 01:52:50 dist-test-slave-gkw7 krb5kdc[11983](info): commencing operation
krb5kdc: starting...
W20260809 01:52:51.231724 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:51 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:52:51.365048 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:46671
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:46671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:51.478669 12002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:51.478938 12002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:51.478979 12002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:51.483009 12002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:51.483068 12002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:51.483088 12002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:51.483104 12002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:51.483119 12002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:51.488368 12002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:46671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:51.489464 12002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:51.490674 12002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:51.499564 12010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:51.499616 12002 server_base.cc:1034] running on GCE node
W20260809 01:52:51.499571 12011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:51.499743 12014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:51.500540 12002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:51.501647 12002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:51.502841 12002 hybrid_clock.cc:648] HybridClock initialized: now 1786240371502813 us; error 54 us; skew 500 ppm
Aug 09 01:52:51 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240371, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:51.509233 12002 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:52:51.510257 12002 webserver.cc:460] Webserver started at http://127.3.220.62:44985/ using document root <none> and password file <none>
I20260809 01:52:51.510795 12002 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:51.510855 12002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:51.511022 12002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:51.513003 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e"
format_stamp: "Formatted at 2026-08-09 01:52:51 on dist-test-slave-gkw7"
server_key: "1af31bc091cbafcdc57c7efed77322a1"
server_key_iv: "7d0eeff4929749500d0a688eaefda5ef"
server_key_version: "encryptionkey@0"
I20260809 01:52:51.513525 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e"
format_stamp: "Formatted at 2026-08-09 01:52:51 on dist-test-slave-gkw7"
server_key: "1af31bc091cbafcdc57c7efed77322a1"
server_key_iv: "7d0eeff4929749500d0a688eaefda5ef"
server_key_version: "encryptionkey@0"
I20260809 01:52:51.517238 12002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:52:51.519610 12021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:51.520829 12002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:51.520970 12002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e"
format_stamp: "Formatted at 2026-08-09 01:52:51 on dist-test-slave-gkw7"
server_key: "1af31bc091cbafcdc57c7efed77322a1"
server_key_iv: "7d0eeff4929749500d0a688eaefda5ef"
server_key_version: "encryptionkey@0"
I20260809 01:52:51.521070 12002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:51.540194 12002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:51.543398 12002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:51.543612 12002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:51.550843 12002 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:46671
I20260809 01:52:51.550868 12098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:46671 every 8 connection(s)
I20260809 01:52:51.551781 12002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:51.552116 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 12002
I20260809 01:52:51.552271 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:51.552558 3952 external_mini_cluster.cc:1428] Setting key 30d931eabbe185e7ef5654d4fd59088b
I20260809 01:52:51.554720 12101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:51.560055 12101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e: Bootstrap starting.
Aug 09 01:52:51 dist-test-slave-gkw7 krb5kdc[11983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:51.562656 12101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:51.563367 12101 log.cc:824] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:51.565600 12101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e: No bootstrap required, opened a new log
I20260809 01:52:51.566931 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:51.554038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.3.220.62:46671)
0809 01:52:51.554442 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:51.554467 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:51.554503 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:52:51.555201 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:51.555218 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:51.555540 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:51.555875 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:51.556696 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:51.557760 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:51.558602 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:51.559066 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:51.559251 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:51.562152 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:51.562158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:51.562171 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:51.562197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:51.564200 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:51.564678 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:51.564682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:51.564687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:51.564787 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:51.565069 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:51.565071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:51.565072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:51.565470 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:51.565708 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:51.565950 (+ 242us) server_negotiation.cc:299] Negotiation successful
0809 01:52:51.566246 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":132,"threads_started":1}
I20260809 01:52:51.569389 12101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46671 } }
I20260809 01:52:51.569643 12101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:51.569736 12101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5ffea75fb44bcf8a4ac23641c2cc5e, State: Initialized, Role: FOLLOWER
I20260809 01:52:51.570135 12101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46671 } }
I20260809 01:52:51.570281 12101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:51.570353 12101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:51.570467 12101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:51.571301 12101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46671 } }
I20260809 01:52:51.571631 12101 leader_election.cc:302] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d5ffea75fb44bcf8a4ac23641c2cc5e; no voters:
I20260809 01:52:51.571941 12101 leader_election.cc:288] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:51.572141 12109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:51.572456 12109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [term 1 LEADER]: Becoming Leader. State: Replica: 6d5ffea75fb44bcf8a4ac23641c2cc5e, State: Running, Role: LEADER
I20260809 01:52:51.572752 12109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46671 } }
I20260809 01:52:51.573179 12101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:51.574395 12113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d5ffea75fb44bcf8a4ac23641c2cc5e. Latest consensus state: current_term: 1 leader_uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46671 } } }
I20260809 01:52:51.574528 12113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:51.574894 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ffea75fb44bcf8a4ac23641c2cc5e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46671 } } }
I20260809 01:52:51.574954 12122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:51.575048 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:51.577812 12122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:51.583184 12122 catalog_manager.cc:1347] Generated new cluster ID: 61de7040c0414856835e4a73a6636812
I20260809 01:52:51.583252 12122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:51.588969 12122 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:51.589922 12122 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:51.595043 12122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d5ffea75fb44bcf8a4ac23641c2cc5e: Generated new TSK 0
I20260809 01:52:51.595644 12122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:52:51.626933 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46671
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:51.741927 12149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:51.742238 12149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:51.742314 12149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:51.746374 12149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:51.746462 12149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:51.746556 12149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:51.751784 12149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:51.752910 12149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:51.754078 12149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:51.763931 12159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:51.763939 12161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:51.764011 12158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:51.764199 12149 server_base.cc:1034] running on GCE node
I20260809 01:52:51.764688 12149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:51.765343 12149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:51.766533 12149 hybrid_clock.cc:648] HybridClock initialized: now 1786240371766519 us; error 26 us; skew 500 ppm
Aug 09 01:52:51 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240371, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:51.769351 12149 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:52:51.770241 12149 webserver.cc:460] Webserver started at http://127.3.220.1:42055/ using document root <none> and password file <none>
I20260809 01:52:51.770812 12149 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:51.770905 12149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:51.771140 12149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:51.773224 12149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "ff91ea5d6d6f4060b31e3ab7ee3eac27"
format_stamp: "Formatted at 2026-08-09 01:52:51 on dist-test-slave-gkw7"
server_key: "1a0e6478125e8f043295b2d112076a67"
server_key_iv: "9ac94b23fb701a11a4006620a5706846"
server_key_version: "encryptionkey@0"
I20260809 01:52:51.773808 12149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "ff91ea5d6d6f4060b31e3ab7ee3eac27"
format_stamp: "Formatted at 2026-08-09 01:52:51 on dist-test-slave-gkw7"
server_key: "1a0e6478125e8f043295b2d112076a67"
server_key_iv: "9ac94b23fb701a11a4006620a5706846"
server_key_version: "encryptionkey@0"
I20260809 01:52:51.777832 12149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 01:52:51.780247 12172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:51.781440 12149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:51.781617 12149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "ff91ea5d6d6f4060b31e3ab7ee3eac27"
format_stamp: "Formatted at 2026-08-09 01:52:51 on dist-test-slave-gkw7"
server_key: "1a0e6478125e8f043295b2d112076a67"
server_key_iv: "9ac94b23fb701a11a4006620a5706846"
server_key_version: "encryptionkey@0"
I20260809 01:52:51.781733 12149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:51.829221 12149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:51.832890 12149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:51.833122 12149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:51.833843 12149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:51.834734 12149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:51.834810 12149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:51.834883 12149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:51.834934 12149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:51.844318 12149 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:35171
I20260809 01:52:51.844403 12332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:35171 every 8 connection(s)
I20260809 01:52:51.845324 12149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:51.854768 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 12149
I20260809 01:52:51.854923 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:51.855234 3952 external_mini_cluster.cc:1428] Setting key 30244e523874a52e18bf98fb382d404d
Aug 09 01:52:51 dist-test-slave-gkw7 krb5kdc[11983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240371, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:51.860881 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:51.847389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:48841 (local address 127.3.220.62:46671)
0809 01:52:51.847564 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:51.847570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:51.848435 (+ 865us) server_negotiation.cc:407] Connection header received
0809 01:52:51.850026 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:51.850029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:51.850077 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:51.850153 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:51.851434 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:51.852030 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:51.853281 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:51.853514 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:51.853651 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:51.856686 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:51.856693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:51.856696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:51.856727 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:51.858520 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:51.859055 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:51.859061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:51.859065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:51.859170 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:51.859661 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:51.859664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:51.859665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:51.859849 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:51.859925 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:51.860633 (+ 708us) server_negotiation.cc:299] Negotiation successful
0809 01:52:51.860759 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 01:52:51.861934 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:51.847785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:46671 (local address 127.3.220.1:48841)
0809 01:52:51.848275 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:51.848321 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:51.849811 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:51.850357 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:51.850364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:51.850769 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:51.851293 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:51.851307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:51.852190 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:51.852194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:51.853167 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:51.853176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:51.853708 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:51.853711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:51.853886 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:51.854526 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:51.854545 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:51.856477 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:51.858650 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:51.858656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:51.858671 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:51.858947 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:51.859300 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:51.859305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:51.859308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:51.859530 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:51.859962 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:51.859972 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:51.860342 (+ 370us) client_negotiation.cc:769] Sending connection context
0809 01:52:51.860616 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 01:52:51.860964 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":107,"threads_started":1}
I20260809 01:52:51.863430 12334 heartbeater.cc:347] Connected to a master server at 127.3.220.62:46671
I20260809 01:52:51.863662 12334 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:51.864173 12334 heartbeater.cc:511] Master 127.3.220.62:46671 requested a full tablet report, sending...
I20260809 01:52:51.865978 12046 ts_manager.cc:194] Registered new tserver with Master: ff91ea5d6d6f4060b31e3ab7ee3eac27 (127.3.220.1:35171)
I20260809 01:52:51.867661 12046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:48841
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:52:51.889998 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46671
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:52.006494 12342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:52.006781 12342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:52.006827 12342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:52.011070 12342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:52.011145 12342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:52.011248 12342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:52.016800 12342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:52.017997 12342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:52.019091 12342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:52.029666 12353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:52.029646 12356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:52.029687 12349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:52.029758 12342 server_base.cc:1034] running on GCE node
I20260809 01:52:52.030339 12342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:52.031044 12342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:52.032238 12342 hybrid_clock.cc:648] HybridClock initialized: now 1786240372032226 us; error 42 us; skew 500 ppm
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.035290 12342 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:52:52.036307 12342 webserver.cc:460] Webserver started at http://127.3.220.2:38207/ using document root <none> and password file <none>
I20260809 01:52:52.036959 12342 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:52.037073 12342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:52.037343 12342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:52.039652 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "7390682db76e43c78e2ccbe08161f31d"
format_stamp: "Formatted at 2026-08-09 01:52:52 on dist-test-slave-gkw7"
server_key: "0aeddc3f78995daad6b8028705add33b"
server_key_iv: "e23d5f9fbbf15f74069feed7867d1377"
server_key_version: "encryptionkey@0"
I20260809 01:52:52.040277 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "7390682db76e43c78e2ccbe08161f31d"
format_stamp: "Formatted at 2026-08-09 01:52:52 on dist-test-slave-gkw7"
server_key: "0aeddc3f78995daad6b8028705add33b"
server_key_iv: "e23d5f9fbbf15f74069feed7867d1377"
server_key_version: "encryptionkey@0"
I20260809 01:52:52.044517 12342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260809 01:52:52.047446 12364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:52.048761 12342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:52:52.048914 12342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "7390682db76e43c78e2ccbe08161f31d"
format_stamp: "Formatted at 2026-08-09 01:52:52 on dist-test-slave-gkw7"
server_key: "0aeddc3f78995daad6b8028705add33b"
server_key_iv: "e23d5f9fbbf15f74069feed7867d1377"
server_key_version: "encryptionkey@0"
I20260809 01:52:52.049038 12342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:52.072981 12342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:52.076664 12342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:52.076844 12342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:52.077412 12342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:52.078455 12342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:52.078511 12342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:52.078557 12342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:52.078583 12342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:52.088917 12342 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:33869
I20260809 01:52:52.089001 12530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:33869 every 8 connection(s)
I20260809 01:52:52.090015 12342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:52.097841 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 12342
I20260809 01:52:52.097942 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:52.098282 3952 external_mini_cluster.cc:1428] Setting key 20c7f61552b37780fc9228ad2f87f911
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:52.105223 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:52.092129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:52291 (local address 127.3.220.62:46671)
0809 01:52:52.092276 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:52.092280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:52.093156 (+ 876us) server_negotiation.cc:407] Connection header received
0809 01:52:52.094649 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:52.094652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:52.094702 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:52.094785 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:52.096126 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.096812 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.098138 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.098421 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.098537 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:52.101079 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:52.101083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:52.101085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:52.101133 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:52.102819 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.103468 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.103470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.103472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.103548 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.103946 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.103949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.103950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.104121 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:52.104236 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:52.104974 (+ 738us) server_negotiation.cc:299] Negotiation successful
0809 01:52:52.105093 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 01:52:52.106516 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:52.092530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:46671 (local address 127.3.220.2:52291)
0809 01:52:52.092995 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:52.093038 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:52.094377 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:52.095028 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:52.095037 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:52.095483 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:52.095975 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:52.095988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.096942 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.096945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:52.097964 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:52.097982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.098579 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.098584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:52.098773 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:52.099364 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:52.099385 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:52.100904 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:52.102938 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:52.102945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:52.102962 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:52.103359 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:52.103643 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:52.103648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:52.103653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:52.103848 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:52.104267 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:52.104276 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:52.104644 (+ 368us) client_negotiation.cc:769] Sending connection context
0809 01:52:52.104884 (+ 240us) client_negotiation.cc:240] Negotiation successful
0809 01:52:52.105294 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260809 01:52:52.107774 12532 heartbeater.cc:347] Connected to a master server at 127.3.220.62:46671
I20260809 01:52:52.108032 12532 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:52.108557 12532 heartbeater.cc:511] Master 127.3.220.62:46671 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:52:52.109699 12046 ts_manager.cc:194] Registered new tserver with Master: 7390682db76e43c78e2ccbe08161f31d (127.3.220.2:33869)
I20260809 01:52:52.110376 12046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:52291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:52:52.129415 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46671
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:52.242713 12538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:52.243002 12538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:52.243073 12538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:52.247109 12538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:52.247189 12538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:52.247278 12538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:52.252449 12538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:52.253536 12538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:52.254688 12538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:52.264496 12544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:52.264607 12538 server_base.cc:1034] running on GCE node
W20260809 01:52:52.264506 12549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:52.264504 12545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:52.265196 12538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:52.265928 12538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:52.267097 12538 hybrid_clock.cc:648] HybridClock initialized: now 1786240372267050 us; error 58 us; skew 500 ppm
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.270061 12538 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:52:52.270931 12538 webserver.cc:460] Webserver started at http://127.3.220.3:36475/ using document root <none> and password file <none>
I20260809 01:52:52.271441 12538 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:52.271500 12538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:52.271695 12538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:52.273952 12538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "a5b4a967baf945d8b1f28ef4987dd1de"
format_stamp: "Formatted at 2026-08-09 01:52:52 on dist-test-slave-gkw7"
server_key: "ab43ce4c9790a32ec6a9077d69a28bde"
server_key_iv: "5f0b7dfb304d03ba252482d907477872"
server_key_version: "encryptionkey@0"
I20260809 01:52:52.274600 12538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b4a967baf945d8b1f28ef4987dd1de"
format_stamp: "Formatted at 2026-08-09 01:52:52 on dist-test-slave-gkw7"
server_key: "ab43ce4c9790a32ec6a9077d69a28bde"
server_key_iv: "5f0b7dfb304d03ba252482d907477872"
server_key_version: "encryptionkey@0"
I20260809 01:52:52.278512 12538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 01:52:52.281045 12556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:52.282393 12538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:52.282545 12538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "a5b4a967baf945d8b1f28ef4987dd1de"
format_stamp: "Formatted at 2026-08-09 01:52:52 on dist-test-slave-gkw7"
server_key: "ab43ce4c9790a32ec6a9077d69a28bde"
server_key_iv: "5f0b7dfb304d03ba252482d907477872"
server_key_version: "encryptionkey@0"
I20260809 01:52:52.282661 12538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:52.311659 12538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:52.315173 12538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:52.315390 12538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:52.316025 12538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:52.316943 12538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:52.317008 12538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:52.317065 12538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:52.317090 12538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:52.326646 12538 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:46549
I20260809 01:52:52.326704 12725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:46549 every 8 connection(s)
I20260809 01:52:52.327580 12538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:52.336874 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 12538
I20260809 01:52:52.337006 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:52.337285 3952 external_mini_cluster.cc:1428] Setting key 8169e466bdba8904ec832d574388a1f4
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:52:52.341686 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:52.329444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:33647 (local address 127.3.220.62:46671)
0809 01:52:52.329653 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:52.329657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:52.330519 (+ 862us) server_negotiation.cc:407] Connection header received
0809 01:52:52.331945 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:52.331949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:52.332004 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:52.332114 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:52.333282 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.333877 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.335035 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.335264 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.335379 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:52.338003 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:52.338007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:52.338010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:52.338043 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:52.339557 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.340115 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.340119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.340121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.340210 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.340554 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.340560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.340563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.340759 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:52.340871 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:52.341360 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 01:52:52.341483 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 01:52:52.342456 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:52.329859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:46671 (local address 127.3.220.3:33647)
0809 01:52:52.330352 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:52.330397 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:52.331726 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:52.332297 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:52.332305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:52.332666 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:52.333122 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:52.333135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.334002 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.334010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:52.334907 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:52.334917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.335417 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.335421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:52.335595 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:52.336153 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:52.336169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:52.337839 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:52.339716 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:52.339722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:52.339736 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:52.340005 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:52.340304 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:52.340308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:52.340310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:52.340455 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:52.340870 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:52.340877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:52.341115 (+ 238us) client_negotiation.cc:769] Sending connection context
0809 01:52:52.341317 (+ 202us) client_negotiation.cc:240] Negotiation successful
0809 01:52:52.341651 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260809 01:52:52.343466 12727 heartbeater.cc:347] Connected to a master server at 127.3.220.62:46671
I20260809 01:52:52.343713 12727 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:52.344182 12727 heartbeater.cc:511] Master 127.3.220.62:46671 requested a full tablet report, sending...
I20260809 01:52:52.345166 12046 ts_manager.cc:194] Registered new tserver with Master: a5b4a967baf945d8b1f28ef4987dd1de (127.3.220.3:46549)
I20260809 01:52:52.345811 12046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:33647
I20260809 01:52:52.351220 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/test-admin.keytab.
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.369635 3952 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 01:52:52.372452 12736 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:52.371012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.3.220.1:35171)
0809 01:52:52.371331 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:52.371340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:52.371372 (+ 32us) server_negotiation.cc:407] Connection header received
0809 01:52:52.371425 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:52.371429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:52.371634 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:52.371760 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:52.372246 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53132: BlockingRecv error: recv got EOF from 127.0.0.1:53132 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":140,"threads_started":1}
W20260809 01:52:52.372707 3952 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5cc)
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.373934 3952 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.375082 3952 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:52.384948 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:52.376028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.3.220.1:35171)
0809 01:52:52.376188 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:52.376192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:52.376206 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:52.376323 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:52.376325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:52.376367 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:52.376437 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:52.376924 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.377779 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.378673 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.379051 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.379134 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:52.381616 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:52.381626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:52.381641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:52.381664 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:52.383273 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.383664 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.383669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.383670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.383748 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.383977 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.383979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.383980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.384223 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:52.384393 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:52.384583 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 01:52:52.384814 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 01:52:52.389019 12736 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:52.388051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.3.220.1:35171)
0809 01:52:52.388157 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:52.388163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:52.388246 (+ 83us) server_negotiation.cc:407] Connection header received
0809 01:52:52.388363 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:52.388367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:52.388411 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:52.388496 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:52.388899 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53150: BlockingRecv error: recv got EOF from 127.0.0.1:53150 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 01:52:52.389143 3952 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5cc' of type FILE
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.390211 3952 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.391072 3952 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:52.399474 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:52.391832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.3.220.1:35171)
0809 01:52:52.391947 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:52.391950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:52.392131 (+ 181us) server_negotiation.cc:407] Connection header received
0809 01:52:52.392201 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:52.392204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:52.392239 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:52.392300 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:52.392746 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.393251 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.394290 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.394504 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.394585 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:52.396599 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:52.396602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:52.396605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:52.396626 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:52.398045 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.398429 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.398432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.398433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.398503 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.398732 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.398734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.398735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.398883 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:52.398995 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:52.399167 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 01:52:52.399295 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 01:52:52.402367 12736 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:52.401180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.3.220.1:35171)
0809 01:52:52.401298 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:52.401302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:52.401444 (+ 142us) server_negotiation.cc:407] Connection header received
0809 01:52:52.401590 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:52.401592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:52.401637 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:52.401701 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:52.402248 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53166: BlockingRecv error: recv got EOF from 127.0.0.1:53166 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 01:52:52.402578 3952 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786240338176169-3952-0/krb5kdc/krb5cc' of type FILE
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.403604 3952 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:52.404475 3952 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[11983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:52:52.412875 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:52.405245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.3.220.1:35171)
0809 01:52:52.405350 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:52.405353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:52.405574 (+ 221us) server_negotiation.cc:407] Connection header received
0809 01:52:52.405728 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:52.405730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:52.405771 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:52.405834 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:52.406309 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.406820 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.407579 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:52.407785 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:52.407858 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:52.409982 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:52.409985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:52.409987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:52.410008 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:52.411258 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.411644 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.411646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.411647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.411718 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:52.411992 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:52.411996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:52.411998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:52.412173 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:52.412394 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:52.412610 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 01:52:52.412718 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 01:52:52.414101 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 12149
I20260809 01:52:52.428122 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 12342
I20260809 01:52:52.439332 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 12538
I20260809 01:52:52.449182 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 12002
2026-08-09T01:52:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2486 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[12764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[12764](info): set up 2 sockets
Aug 09 01:52:52 dist-test-slave-gkw7 krb5kdc[12764](info): commencing operation
krb5kdc: starting...
W20260809 01:52:53.737780 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:52:53 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:52:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.62.keytab.
Entry for principal oryx/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.62.keytab.
I20260809 01:52:53.868642 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:45149
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.62.keytab
--principal=oryx/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:53.986215 12780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:53.986531 12780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:53.986613 12780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:53.990727 12780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:53.990811 12780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:53.990849 12780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:53.990912 12780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:53.990948 12780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:53.996369 12780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.62.keytab
--principal=oryx/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:53.997503 12780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:53.998636 12780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:54.008010 12785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.008010 12789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.008021 12786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:54.008317 12780 server_base.cc:1034] running on GCE node
I20260809 01:52:54.009012 12780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:54.010262 12780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:54.011472 12780 hybrid_clock.cc:648] HybridClock initialized: now 1786240374011453 us; error 28 us; skew 500 ppm
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:54.014603 12780 init.cc:377] Logged in from keytab as oryx/127.3.220.62@KRBTEST.COM (short username oryx)
I20260809 01:52:54.015527 12780 webserver.cc:460] Webserver started at http://127.3.220.62:40139/ using document root <none> and password file <none>
I20260809 01:52:54.016256 12780 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:54.016366 12780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:54.016637 12780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:54.018859 12780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "25c42ff3cd264b7082ea27fa4a0b42ef"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "83a8e63fbe433d9d1c19640bb2b1c110"
server_key_iv: "cb209567b776b1ce927226207642fb73"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.019501 12780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "25c42ff3cd264b7082ea27fa4a0b42ef"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "83a8e63fbe433d9d1c19640bb2b1c110"
server_key_iv: "cb209567b776b1ce927226207642fb73"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.023639 12780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:52:54.026289 12798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.027494 12780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:54.027642 12780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "25c42ff3cd264b7082ea27fa4a0b42ef"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "83a8e63fbe433d9d1c19640bb2b1c110"
server_key_iv: "cb209567b776b1ce927226207642fb73"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.027772 12780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:54.055821 12780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:54.059393 12780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:54.059612 12780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:54.066942 12780 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:45149
I20260809 01:52:54.067243 12873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:45149 every 8 connection(s)
I20260809 01:52:54.067968 12780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:52:54.070997 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:54.076624 12875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: Bootstrap starting.
I20260809 01:52:54.076968 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 12780
I20260809 01:52:54.077118 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:52:54.077412 3952 external_mini_cluster.cc:1428] Setting key a982cc15946917b736334e21989beb3a
I20260809 01:52:54.079336 12875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:54.080084 12875 log.cc:824] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:54.082371 12875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: No bootstrap required, opened a new log
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:52:54.086959 12875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } }
I20260809 01:52:54.087221 12875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:54.087292 12875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c42ff3cd264b7082ea27fa4a0b42ef, State: Initialized, Role: FOLLOWER
I20260809 01:52:54.087814 12875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } }
I20260809 01:52:54.087997 12875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:52:54.088073 12875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:52:54.088205 12875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:54.089399 12875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } }
I20260809 01:52:54.089783 12875 leader_election.cc:302] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25c42ff3cd264b7082ea27fa4a0b42ef; no voters:
I20260809 01:52:54.090157 12875 leader_election.cc:288] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:52:54.090261 12886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:54.090488 12886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 1 LEADER]: Becoming Leader. State: Replica: 25c42ff3cd264b7082ea27fa4a0b42ef, State: Running, Role: LEADER
I20260809 01:52:54.090796 12886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } }
I20260809 01:52:54.091539 12875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:54.092079 12888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c42ff3cd264b7082ea27fa4a0b42ef. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } }
I20260809 01:52:54.092196 12888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:54.092554 12887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } }
I20260809 01:52:54.092662 12887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:54.092934 12891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:54.093447 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:54.078769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.3.220.62:45149)
0809 01:52:54.079191 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:54.079206 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:54.079242 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:52:54.079916 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:54.079935 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:54.080309 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:54.080661 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:54.081506 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.082609 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.083746 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.084202 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.084324 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:54.087227 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:54.087237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:54.087256 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:54.087299 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:54.089380 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:54.089884 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:54.089893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:54.089900 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:54.090042 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:54.090410 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:54.090417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:54.090421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:54.090993 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:54.091205 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:54.091891 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 01:52:54.092200 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":148,"threads_started":1}
I20260809 01:52:54.096086 12891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:54.101893 12891 catalog_manager.cc:1347] Generated new cluster ID: b4288719cdb2436b861f8e28027d272e
I20260809 01:52:54.101966 12891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:54.119166 12891 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:52:54.120499 12891 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:54.132589 12891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: Generated new TSK 0
I20260809 01:52:54.133452 12891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab.
Entry for principal oryx/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab.
I20260809 01:52:54.172725 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45149
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab
--principal=oryx/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:54.289961 12914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:54.290256 12914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:54.290300 12914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:54.294566 12914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:54.294631 12914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:54.294693 12914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:54.299983 12914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab
--principal=oryx/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:54.301057 12914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:54.302117 12914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:52:54.311862 12914 server_base.cc:1034] running on GCE node
W20260809 01:52:54.311825 12921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.311825 12923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.312070 12920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:54.312561 12914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:54.313223 12914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:54.314419 12914 hybrid_clock.cc:648] HybridClock initialized: now 1786240374314371 us; error 55 us; skew 500 ppm
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:54.317251 12914 init.cc:377] Logged in from keytab as oryx/127.3.220.1@KRBTEST.COM (short username oryx)
I20260809 01:52:54.318186 12914 webserver.cc:460] Webserver started at http://127.3.220.1:34437/ using document root <none> and password file <none>
I20260809 01:52:54.318770 12914 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:54.318866 12914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:54.319108 12914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:54.321156 12914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "c782dcddc5764bd39e69380cc3ad640a"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "1deada40ec7280124496223136ad2f77"
server_key_iv: "769ee724d4745b2d4fec898ac41ec5ba"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.321792 12914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "c782dcddc5764bd39e69380cc3ad640a"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "1deada40ec7280124496223136ad2f77"
server_key_iv: "769ee724d4745b2d4fec898ac41ec5ba"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.325646 12914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 01:52:54.328234 12933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.329491 12914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:54.329648 12914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "c782dcddc5764bd39e69380cc3ad640a"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "1deada40ec7280124496223136ad2f77"
server_key_iv: "769ee724d4745b2d4fec898ac41ec5ba"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.329757 12914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:54.351543 12914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:54.354449 12914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:54.354607 12914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:54.355152 12914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:54.355954 12914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:54.355999 12914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.356040 12914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:54.356061 12914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.364975 12914 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:37357
I20260809 01:52:54.365002 13091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:37357 every 8 connection(s)
I20260809 01:52:54.365975 12914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:54.370213 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 12914
I20260809 01:52:54.370342 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:52:54.370653 3952 external_mini_cluster.cc:1428] Setting key 37c0f06ac658aa386ebc081b1c87055d
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:52:54.381433 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:54.367833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:47187 (local address 127.3.220.62:45149)
0809 01:52:54.367975 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:54.367979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:54.368758 (+ 779us) server_negotiation.cc:407] Connection header received
0809 01:52:54.370159 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:54.370162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:54.370253 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:54.370388 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:54.371944 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.372558 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.373787 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.374047 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.374162 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:54.377003 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:54.377010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:54.377016 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:54.377052 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:54.378926 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:54.379553 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:54.379557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:54.379558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:54.379656 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:54.380076 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:54.380083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:54.380085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:54.380367 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:54.380484 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:54.381038 (+ 554us) server_negotiation.cc:299] Negotiation successful
0809 01:52:54.381209 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:52:54.382380 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:54.368151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:47187)
0809 01:52:54.368595 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:54.368637 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:54.369929 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:54.370639 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:54.370647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:54.371153 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:54.371765 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:54.371784 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.372708 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.372711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:54.373655 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:54.373673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.374250 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.374259 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:54.374403 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:54.375026 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:54.375042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:54.376812 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:54.379078 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:54.379084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:54.379099 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:54.379438 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:54.379782 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:54.379785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:54.379787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:54.379955 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:54.380503 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:54.380509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:54.380772 (+ 263us) client_negotiation.cc:769] Sending connection context
0809 01:52:54.380943 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 01:52:54.381233 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":124,"threads_started":1}
I20260809 01:52:54.383782 13092 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45149
I20260809 01:52:54.384003 13092 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:54.384518 13092 heartbeater.cc:511] Master 127.3.220.62:45149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.2@KRBTEST.COM" created.
I20260809 01:52:54.386174 12818 ts_manager.cc:194] Registered new tserver with Master: c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357)
I20260809 01:52:54.387916 12818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:47187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab.
Entry for principal oryx/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab.
I20260809 01:52:54.400938 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45149
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab
--principal=oryx/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:54.512603 13099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:54.512848 13099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:54.512885 13099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:54.516799 13099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:54.516850 13099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:54.516958 13099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:54.522276 13099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab
--principal=oryx/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:54.523303 13099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:54.524318 13099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:54.534042 13108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.534085 13106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.534116 13111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:54.534572 13099 server_base.cc:1034] running on GCE node
I20260809 01:52:54.534920 13099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:54.535522 13099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:54.536690 13099 hybrid_clock.cc:648] HybridClock initialized: now 1786240374536687 us; error 35 us; skew 500 ppm
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:54.539399 13099 init.cc:377] Logged in from keytab as oryx/127.3.220.2@KRBTEST.COM (short username oryx)
I20260809 01:52:54.540282 13099 webserver.cc:460] Webserver started at http://127.3.220.2:38391/ using document root <none> and password file <none>
I20260809 01:52:54.540813 13099 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:54.540872 13099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:54.541033 13099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:54.543071 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "39f5e2bf2e0543a88e3d3d6783756314"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "b60af748fa870c80723bbf87685b1ace"
server_key_iv: "85c33c2c48baf443b48f81a123ab08c6"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.543596 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "39f5e2bf2e0543a88e3d3d6783756314"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "b60af748fa870c80723bbf87685b1ace"
server_key_iv: "85c33c2c48baf443b48f81a123ab08c6"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.547353 13099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:52:54.549806 13122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.551023 13099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:54.551115 13099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "39f5e2bf2e0543a88e3d3d6783756314"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "b60af748fa870c80723bbf87685b1ace"
server_key_iv: "85c33c2c48baf443b48f81a123ab08c6"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.551211 13099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:54.566485 13099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:54.569453 13099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:54.569667 13099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:54.570204 13099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:54.570993 13099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:54.571038 13099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.571076 13099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:54.571092 13099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.579990 13099 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:43743
I20260809 01:52:54.580018 13281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:43743 every 8 connection(s)
I20260809 01:52:54.580879 13099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:54.588398 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 13099
I20260809 01:52:54.588538 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:52:54.588843 3952 external_mini_cluster.cc:1428] Setting key 9c20dd62d0ad26aa581195ad427130e4
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:52:54.596403 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:54.582817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:56019 (local address 127.3.220.62:45149)
0809 01:52:54.582961 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:54.582965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:54.583728 (+ 763us) server_negotiation.cc:407] Connection header received
0809 01:52:54.585387 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:54.585391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:54.585478 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:54.585610 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:54.586763 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.587308 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.588618 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.588859 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.588938 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:54.592040 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:54.592044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:54.592047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:54.592078 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:54.593950 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:54.594640 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:54.594647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:54.594651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:54.594758 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:54.595103 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:54.595109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:54.595114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:54.595381 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:54.595493 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:54.595993 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 01:52:54.596153 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:52:54.597076 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:54.583131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:56019)
0809 01:52:54.583579 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:54.583621 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:54.585062 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:54.585803 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:54.585810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:54.586155 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:54.586622 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:54.586635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.587421 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.587423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:54.588486 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:54.588499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.589031 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.589036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:54.589255 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:54.589997 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:54.590013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:54.591864 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:54.594068 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:54.594077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:54.594096 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:54.594491 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:54.594861 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:54.594864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:54.594865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:54.595002 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:54.595489 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:54.595494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:54.595733 (+ 239us) client_negotiation.cc:769] Sending connection context
0809 01:52:54.595892 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 01:52:54.596143 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260809 01:52:54.598325 13283 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45149
I20260809 01:52:54.598613 13283 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:54.599218 13283 heartbeater.cc:511] Master 127.3.220.62:45149 requested a full tablet report, sending...
I20260809 01:52:54.600343 12818 ts_manager.cc:194] Registered new tserver with Master: 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743)
I20260809 01:52:54.600925 12818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:56019
WARNING: no policy specified for oryx/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab.
Entry for principal oryx/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab.
I20260809 01:52:54.620620 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45149
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab
--principal=oryx/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:54.736140 13290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:54.736424 13290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:54.736505 13290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:54.740546 13290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:54.740624 13290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:54.740708 13290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:54.745904 13290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab
--principal=oryx/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:54.746893 13290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:54.747874 13290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:54.757614 13303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:54.757794 13290 server_base.cc:1034] running on GCE node
W20260809 01:52:54.757603 13305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.757603 13301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:54.758339 13290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:54.758986 13290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:54.760174 13290 hybrid_clock.cc:648] HybridClock initialized: now 1786240374760155 us; error 30 us; skew 500 ppm
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:54.762981 13290 init.cc:377] Logged in from keytab as oryx/127.3.220.3@KRBTEST.COM (short username oryx)
I20260809 01:52:54.763859 13290 webserver.cc:460] Webserver started at http://127.3.220.3:38941/ using document root <none> and password file <none>
I20260809 01:52:54.764441 13290 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:54.764533 13290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:54.764775 13290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:54.766836 13290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "fd69de1d46fe4e24a3726b915c893e05"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "6c342fd903c2a35ba1ae93668246109c"
server_key_iv: "00bd7acbb0ad8471dc276eea9a5ec467"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.767460 13290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "fd69de1d46fe4e24a3726b915c893e05"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "6c342fd903c2a35ba1ae93668246109c"
server_key_iv: "00bd7acbb0ad8471dc276eea9a5ec467"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.771174 13290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260809 01:52:54.773627 13314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.774847 13290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:54.774983 13290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "fd69de1d46fe4e24a3726b915c893e05"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "6c342fd903c2a35ba1ae93668246109c"
server_key_iv: "00bd7acbb0ad8471dc276eea9a5ec467"
server_key_version: "encryptionkey@0"
I20260809 01:52:54.775090 13290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:54.788631 13290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:54.791656 13290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:54.791913 13290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:54.792543 13290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:54.793438 13290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:54.793486 13290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.793606 13290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:54.793637 13290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:54.803010 13290 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:36817
I20260809 01:52:54.803036 13473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:36817 every 8 connection(s)
I20260809 01:52:54.803897 13290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:54.807859 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 13290
I20260809 01:52:54.808022 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:52:54.808405 3952 external_mini_cluster.cc:1428] Setting key 461e05f329e889718b84b94ca86c3ab6
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:52:54.819015 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:54.806088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:48853 (local address 127.3.220.62:45149)
0809 01:52:54.806298 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:54.806303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:54.807497 (+ 1194us) server_negotiation.cc:407] Connection header received
0809 01:52:54.809303 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:54.809305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:54.809362 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:54.809481 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:54.810978 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.811523 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.812663 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.812902 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.813011 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:54.815579 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:54.815583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:54.815586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:54.815613 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:54.816911 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:54.817431 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:54.817435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:54.817436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:54.817561 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:54.817884 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:54.817889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:54.817891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:54.818123 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:54.818233 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:54.818717 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 01:52:54.818843 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 01:52:54.819769 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:54.806583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:48853)
0809 01:52:54.807301 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:54.807383 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:54.809113 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:54.809676 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:54.809686 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:54.810197 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:54.810817 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:54.810838 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.811628 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.811631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:54.812531 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:54.812541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:54.813049 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:54.813053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:54.813255 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:54.813860 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:54.813876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:54.815417 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:54.817032 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:54.817038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:54.817052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:54.817322 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:54.817641 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:54.817644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:54.817646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:54.817786 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:54.818233 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:54.818241 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:54.818478 (+ 237us) client_negotiation.cc:769] Sending connection context
0809 01:52:54.818666 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 01:52:54.818977 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":115,"threads_started":1}
I20260809 01:52:54.820902 13474 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45149
I20260809 01:52:54.821159 13474 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:54.821738 13474 heartbeater.cc:511] Master 127.3.220.62:45149 requested a full tablet report, sending...
I20260809 01:52:54.822736 12818 ts_manager.cc:194] Registered new tserver with Master: fd69de1d46fe4e24a3726b915c893e05 (127.3.220.3:36817)
I20260809 01:52:54.823302 12818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:48853
I20260809 01:52:54.834497 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.220.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.61.keytab.
Entry for principal oryx/127.3.220.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.61.keytab.
I20260809 01:52:54.864511 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.61:41143
--webserver_interface=127.3.220.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.61.keytab
--principal=oryx/127.3.220.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:45149,127.3.220.61:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:52:54.977725 13482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:54.978012 13482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:54.978085 13482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:54.982173 13482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:54.982260 13482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:54.982298 13482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:54.982350 13482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:54.982388 13482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:54.987566 13482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:45149,127.3.220.61:41143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.61.keytab
--principal=oryx/127.3.220.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.61:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.220.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
W20260809 01:52:54.987816 13482 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.220.62:45149,127.3.220.61:41143'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 01:52:54.988713 13482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:54.989795 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:54.998982 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.998986 13495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:54.998986 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:54.999266 13482 server_base.cc:1034] running on GCE node
I20260809 01:52:54.999902 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:55.001204 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:55.002410 13482 hybrid_clock.cc:648] HybridClock initialized: now 1786240375002403 us; error 60 us; skew 500 ppm
Aug 09 01:52:54 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:55.005504 13482 init.cc:377] Logged in from keytab as oryx/127.3.220.61@KRBTEST.COM (short username oryx)
I20260809 01:52:55.006484 13482 webserver.cc:460] Webserver started at http://127.3.220.61:33709/ using document root <none> and password file <none>
I20260809 01:52:55.007081 13482 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:55.007155 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:55.007339 13482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:55.009629 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data/instance:
uuid: "8208ee9ce1d243faa7ee08a8c7177ada"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "ac707b0eaaf1be91fb7572b87b931f78"
server_key_iv: "e83f7304349bcec60e2c5079558a9bd4"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.010222 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal/instance:
uuid: "8208ee9ce1d243faa7ee08a8c7177ada"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "ac707b0eaaf1be91fb7572b87b931f78"
server_key_iv: "e83f7304349bcec60e2c5079558a9bd4"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.014288 13482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 01:52:55.016803 13503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.018226 13482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:52:55.018402 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
uuid: "8208ee9ce1d243faa7ee08a8c7177ada"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "ac707b0eaaf1be91fb7572b87b931f78"
server_key_iv: "e83f7304349bcec60e2c5079558a9bd4"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.018497 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:55.053658 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:55.057333 13482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:55.057497 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:55.065122 13482 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.61:41143
I20260809 01:52:55.065434 13578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.61:41143 every 8 connection(s)
I20260809 01:52:55.066174 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
I20260809 01:52:55.069064 13580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:55.071463 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 13482
I20260809 01:52:55.073402 13580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.220.61@KRBTEST.COM
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.61@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:52:55.086417 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.075895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.3.220.62:45149)
0809 01:52:55.076057 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.076063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.076395 (+ 332us) server_negotiation.cc:407] Connection header received
0809 01:52:55.076509 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.076512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.076566 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.076656 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.077488 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.078325 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.079509 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.079793 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.079906 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.082982 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.082985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.082987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.083010 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.084529 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.085018 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.085020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.085021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.085095 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.085367 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.085369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.085371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.085643 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.085745 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.086159 (+ 414us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.086274 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:52:55.087204 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.075383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:34600)
0809 01:52:55.076259 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.076305 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.076415 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.076716 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.076722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.077128 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.077330 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.077341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.078439 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.078442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.079365 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.079374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.079948 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.079952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.080070 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.080619 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.080636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.082830 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.084629 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.084637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.084642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.084920 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.085187 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.085190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.085192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.085278 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.085756 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.085761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.085922 (+ 161us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.086573 (+ 651us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.086761 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":159,"threads_started":1}
I20260809 01:52:55.087205 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.073092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.3.220.61:41143)
0809 01:52:55.073556 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.073578 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.073622 (+ 44us) server_negotiation.cc:407] Connection header received
0809 01:52:55.074429 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.074452 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.074850 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.075209 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.076117 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.077347 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.078355 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.079075 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.079330 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.082369 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.082378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.082391 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.082418 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.084016 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.084478 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.084490 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.084494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.084628 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.084916 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.084920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.084921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.085454 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.085672 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.085929 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.086234 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":283,"spinlock_wait_cycles":2816,"thread_start_us":107,"threads_started":1}
I20260809 01:52:55.088564 13580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.61@KRBTEST.COM for oryx/127.3.220.61@KRBTEST.COM
I20260809 01:52:55.096529 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.089097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.0.0.1:49570)
0809 01:52:55.089255 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.089267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.089357 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.089649 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.089651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.089787 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.089881 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.089885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.090983 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.090985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.091542 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.091550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.091996 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.091999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.092155 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.092567 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.092578 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.093885 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.095286 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.095289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.095290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.095482 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.095728 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.095731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.095732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.095815 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.096121 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.096123 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.096219 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.096299 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.096389 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":37}
I20260809 01:52:55.096614 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.089169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.3.220.61:41143)
0809 01:52:55.089300 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.089305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.089320 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:52:55.089445 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.089447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.089492 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.089628 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.089977 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.090887 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.091668 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.091890 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.091998 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.093998 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.094001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.094003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.094022 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.095192 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.095571 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.095573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.095574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.095644 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.095892 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.095894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.095895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.096043 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.096115 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.096358 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.096446 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":51}
I20260809 01:52:55.099365 13580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Bootstrap starting.
I20260809 01:52:55.101668 13580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:55.102370 13580 log.cc:824] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:55.104197 13580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: No bootstrap required, opened a new log
I20260809 01:52:55.107349 13580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } }
I20260809 01:52:55.107661 13580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:55.107724 13580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8208ee9ce1d243faa7ee08a8c7177ada, State: Initialized, Role: FOLLOWER
I20260809 01:52:55.108161 13580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } }
I20260809 01:52:55.108613 13592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } } }
I20260809 01:52:55.108736 13592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:52:55.109340 13580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: configured and running, proceeding with master startup.
W20260809 01:52:55.114135 13607 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:52:55.114218 13607 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:52:55.115916 13482 master_runner.cc:428] Detected that this master 8208ee9ce1d243faa7ee08a8c7177ada is joining an existing cluster
I20260809 01:52:55.115995 13482 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.220.61:41143
I20260809 01:52:55.127220 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.118157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.3.220.61:41143)
0809 01:52:55.118259 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.118263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.118761 (+ 498us) server_negotiation.cc:407] Connection header received
0809 01:52:55.118854 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.118857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.118898 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.118980 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.119515 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.120306 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.121479 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.121874 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.121990 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.123599 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.123606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.123610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.123633 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.125408 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.126006 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.126012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.126017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.126098 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.126382 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.126387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.126389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.126557 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.126662 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.126926 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.127056 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":768}
I20260809 01:52:55.127393 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.118115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.3.220.62:45149)
0809 01:52:55.118246 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.118250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.118597 (+ 347us) server_negotiation.cc:407] Connection header received
0809 01:52:55.118836 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.118840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.118895 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.118983 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.119671 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.120432 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.121304 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.121637 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.121744 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.123650 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.123654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.123659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.123699 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.125490 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.126059 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.126065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.126067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.126160 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.126477 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.126480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.126482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.126704 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.126775 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.127117 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.127238 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 01:52:55.127233 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.118260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.0.0.1:49586)
0809 01:52:55.118661 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.118684 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.118770 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.118994 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.118996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.119292 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.119406 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.119412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.120427 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.120431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.121367 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.121378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.121991 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.121994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.122099 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.123129 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.123143 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.123471 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.125553 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.125557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.125559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.125899 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.126200 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.126204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.126206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.126293 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.126650 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.126652 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.126756 (+ 104us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.126860 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.127007 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260809 01:52:55.127924 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.118041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:34604)
0809 01:52:55.118495 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.118513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.118675 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.119110 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.119114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.119381 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.119520 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.119528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.120545 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.120550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.121172 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.121187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.121764 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.121768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.121943 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.122516 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.122536 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.122978 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.125618 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.125621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.125623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.125912 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.126271 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.126274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.126276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.126372 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.126849 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.126852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.126939 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.127595 (+ 656us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.127745 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":56,"spinlock_wait_cycles":2432,"thread_start_us":123,"threads_started":1}
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.62@KRBTEST.COM for oryx/127.3.220.61@KRBTEST.COM
I20260809 01:52:55.141253 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.132477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.0.0.1:49590)
0809 01:52:55.132866 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.132922 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.133062 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.133571 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.133576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.133920 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.134016 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.134022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.134868 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.134872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.135505 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.135513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.135967 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.135971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.136083 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.136568 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.136579 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.137913 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.139478 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.139489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.139494 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.139728 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.140036 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.140039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.140041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.140142 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.140602 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.140610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.140741 (+ 131us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.140894 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.141084 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260809 01:52:55.141609 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.133209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.3.220.61:41143)
0809 01:52:55.133324 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.133327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.133338 (+ 11us) server_negotiation.cc:407] Connection header received
0809 01:52:55.133388 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.133390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.133427 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.133516 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.134264 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.134756 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.135630 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.135843 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.135963 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.138048 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.138052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.138056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.138081 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.139365 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.139855 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.139859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.139861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.139947 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.140245 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.140249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.140251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.140484 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.140587 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.140986 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.141100 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:52:55.142910 12818 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.220.61:41143: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada"
member_type: NON_VOTER
last_known_addr {
host: "127.3.220.61"
port: 41143
}
attrs {
promote: true
}
}
dest_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef"
cas_config_opid_index: -1
I20260809 01:52:55.143638 12818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } }
I20260809 01:52:55.145506 13619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8208ee9ce1d243faa7ee08a8c7177ada (127.3.220.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } }
I20260809 01:52:55.146180 13541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:55.147318 13619 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.220.61:41143
I20260809 01:52:55.147559 13619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } }
I20260809 01:52:55.147641 13619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:55.147645 13541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Refusing update from remote peer 25c42ff3cd264b7082ea27fa4a0b42ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 01:52:55.148087 13482 master.cc:537] Master@127.3.220.61:41143 shutting down...
I20260809 01:52:55.148476 13619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 01:52:55.150117 12803 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 01:52:55.151172 12803 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef -> Peer 8208ee9ce1d243faa7ee08a8c7177ada (127.3.220.61:41143): Couldn't send request to peer 8208ee9ce1d243faa7ee08a8c7177ada. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 01:52:55.153349 13482 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:55.153656 13482 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:55.153756 13482 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: stopping tablet replica
I20260809 01:52:55.159103 13482 master.cc:552] Master@127.3.220.61:41143 shutdown complete.
I20260809 01:52:55.159160 13482 master_runner.cc:305] Clearing existing system tablet
I20260809 01:52:55.159425 13482 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:55.159493 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
WARNING: no policy specified for oryx/127.3.220.60@KRBTEST.COM; defaulting to no policy
I20260809 01:52:55.160996 13482 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260809 01:52:55.161996 13626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.162235 13482 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:55.162395 13482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:55.162439 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
uuid: "8208ee9ce1d243faa7ee08a8c7177ada"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "ac707b0eaaf1be91fb7572b87b931f78"
server_key_iv: "e83f7304349bcec60e2c5079558a9bd4"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.60@KRBTEST.COM" created.
I20260809 01:52:55.166267 13482 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:55.169341 13482 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 01:52:55.169450 13482 log.cc:1196] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 01:52:55.169855 13482 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Deleting consensus metadata
I20260809 01:52:55.170059 13482 master_runner.cc:315] Copying system tablet from 127.3.220.62:45149
I20260809 01:52:55.171563 13482 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: tablet copy: Beginning tablet copy session from remote peer at address 127.3.220.62:45149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.60.keytab.
Entry for principal oryx/127.3.220.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.60.keytab.
I20260809 01:52:55.180578 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.172218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:34616)
0809 01:52:55.172657 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.172675 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.172837 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.173192 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.173196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.173440 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.173626 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.173637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.174746 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.174748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.175335 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.175343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.175950 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.175953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.176038 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.176558 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.176570 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.176892 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.178900 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.178904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.178906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.179126 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.179482 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.179485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.179487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.179576 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.179992 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.179996 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.180104 (+ 108us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.180224 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.180384 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":99,"threads_started":1}
I20260809 01:52:55.180567 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.172345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.3.220.62:45149)
0809 01:52:55.172498 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.172503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.172801 (+ 298us) server_negotiation.cc:407] Connection header received
0809 01:52:55.172969 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.172973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.173050 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.173146 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.173768 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.174596 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.175450 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.175805 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.175934 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.177057 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.177061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.177064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.177097 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.178766 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.179246 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.179249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.179251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.179338 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.179676 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.179680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.179684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.179892 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.180002 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.180282 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.180398 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 01:52:55.181203 12853 tablet_copy_service.cc:140] P 25c42ff3cd264b7082ea27fa4a0b42ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8208ee9ce1d243faa7ee08a8c7177ada ({username='oryx', principal='oryx/127.3.220.61@KRBTEST.COM'} at 127.0.0.1:34616)
I20260809 01:52:55.181447 12853 tablet_copy_service.cc:161] P 25c42ff3cd264b7082ea27fa4a0b42ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8208ee9ce1d243faa7ee08a8c7177ada at {username='oryx', principal='oryx/127.3.220.61@KRBTEST.COM'} at 127.0.0.1:34616: session id = 8208ee9ce1d243faa7ee08a8c7177ada-00000000000000000000000000000000
I20260809 01:52:55.182041 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.60:36917
--webserver_interface=127.3.220.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.60.keytab
--principal=oryx/127.3.220.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:45149,127.3.220.61:41143,127.3.220.60:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:52:55.183981 12853 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 01:52:55.185905 13482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 01:52:55.189019 13482 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: tablet copy: Starting download of 0 data blocks...
I20260809 01:52:55.189249 13482 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: tablet copy: Starting download of 1 WAL segments...
I20260809 01:52:55.191303 13482 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 01:52:55.193663 12853 tablet_copy_service.cc:342] P 25c42ff3cd264b7082ea27fa4a0b42ef: Request end of tablet copy session 8208ee9ce1d243faa7ee08a8c7177ada-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.220.61@KRBTEST.COM'} at 127.0.0.1:34616
I20260809 01:52:55.193796 12853 tablet_copy_service.cc:433] P 25c42ff3cd264b7082ea27fa4a0b42ef: ending tablet copy session 8208ee9ce1d243faa7ee08a8c7177ada-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8208ee9ce1d243faa7ee08a8c7177ada
I20260809 01:52:55.201656 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:55.203183 13639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:55.203665 13482 server_base.cc:1034] running on GCE node
W20260809 01:52:55.203802 13642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.205745 13640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:55.206010 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:55.206342 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:55.207548 13482 hybrid_clock.cc:648] HybridClock initialized: now 1786240375207536 us; error 31 us; skew 500 ppm
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:55.209175 13482 init.cc:377] Logged in from keytab as oryx/127.3.220.61@KRBTEST.COM (short username oryx)
I20260809 01:52:55.209683 13482 webserver.cc:460] Webserver started at http://127.3.220.61:34675/ using document root <none> and password file <none>
I20260809 01:52:55.209877 13482 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:55.209923 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:55.211355 13482 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 01:52:55.212378 13673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.212638 13482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 01:52:55.212711 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
uuid: "8208ee9ce1d243faa7ee08a8c7177ada"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "ac707b0eaaf1be91fb7572b87b931f78"
server_key_iv: "e83f7304349bcec60e2c5079558a9bd4"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.212780 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:55.225879 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:55.228120 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:55.233574 13482 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.61:41143
I20260809 01:52:55.234230 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
I20260809 01:52:55.236197 13752 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:52:55.237260 13752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Bootstrap starting.
I20260809 01:52:55.245671 13751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.61:41143 every 8 connection(s)
I20260809 01:52:55.248626 13752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:52:55.248970 13752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Bootstrap complete.
I20260809 01:52:55.249619 13752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } }
I20260809 01:52:55.249775 13752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8208ee9ce1d243faa7ee08a8c7177ada, State: Initialized, Role: LEARNER
I20260809 01:52:55.249913 13752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } }
I20260809 01:52:55.250411 13752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:55.250662 13755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } }
I20260809 01:52:55.250854 13755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: This master's current role is: LEARNER
I20260809 01:52:55.253888 13767 catalog_manager.cc:1259] Loaded cluster ID: b4288719cdb2436b861f8e28027d272e
I20260809 01:52:55.253984 13767 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: loading cluster ID for follower catalog manager: success
I20260809 01:52:55.256157 13767 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: acquiring CA information for follower catalog manager: success
I20260809 01:52:55.257462 13767 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 01:52:55.318656 13638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:55.318997 13638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:55.319067 13638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:55.323844 13638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:52:55.323896 13638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:55.323915 13638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:55.323952 13638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:52:55.323980 13638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:52:55.329171 13638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:45149,127.3.220.61:41143,127.3.220.60:36917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.60.keytab
--principal=oryx/127.3.220.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.60:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.220.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:55.330330 13638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:55.331512 13638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:55.340216 13776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.340260 13777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.340260 13779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:55.340582 13638 server_base.cc:1034] running on GCE node
I20260809 01:52:55.341069 13638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:55.342218 13638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:55.343405 13638 hybrid_clock.cc:648] HybridClock initialized: now 1786240375343390 us; error 26 us; skew 500 ppm
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:55.346087 13638 init.cc:377] Logged in from keytab as oryx/127.3.220.60@KRBTEST.COM (short username oryx)
I20260809 01:52:55.346916 13638 webserver.cc:460] Webserver started at http://127.3.220.60:34395/ using document root <none> and password file <none>
I20260809 01:52:55.347465 13638 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:55.347559 13638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:55.347828 13638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:52:55.349985 13638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data/instance:
uuid: "419b6368ced541299d700c2059e48866"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "f7e915ef276b71e6089663dbed0ba2ff"
server_key_iv: "7231b7ad69da06082bcf9789b45637cc"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.350575 13638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal/instance:
uuid: "419b6368ced541299d700c2059e48866"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "f7e915ef276b71e6089663dbed0ba2ff"
server_key_iv: "7231b7ad69da06082bcf9789b45637cc"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.354338 13638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:52:55.356909 13792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.358180 13638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:55.358314 13638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
uuid: "419b6368ced541299d700c2059e48866"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "f7e915ef276b71e6089663dbed0ba2ff"
server_key_iv: "7231b7ad69da06082bcf9789b45637cc"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.358414 13638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:55.373057 13638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:55.375998 13638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:55.376214 13638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:55.383697 13638 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.60:36917
I20260809 01:52:55.383754 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.60:36917 every 8 connection(s)
I20260809 01:52:55.384701 13638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
I20260809 01:52:55.387555 13872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:55.390354 13092 heartbeater.cc:503] Master 127.3.220.62:45149 was elected leader, sending a full tablet report...
I20260809 01:52:55.390839 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 13638
I20260809 01:52:55.391768 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.60@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.220.60@KRBTEST.COM
I20260809 01:52:55.404482 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.393910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.3.220.62:45149)
0809 01:52:55.394038 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.394042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.394315 (+ 273us) server_negotiation.cc:407] Connection header received
0809 01:52:55.394585 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.394588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.394638 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.394714 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.395644 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.396193 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.397218 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.397460 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.397590 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.400297 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.400300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.400302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.400325 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.402157 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.402719 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.402722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.402724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.402820 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.403289 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.403292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.403293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.403545 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.403619 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.404187 (+ 568us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.404318 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 01:52:55.405310 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.393773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:34620)
0809 01:52:55.394193 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.394232 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.394400 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.394777 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.394783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.395155 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.395506 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.395521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.396352 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.396355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.397091 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.397100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.397620 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.397624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.397785 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.398456 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.398480 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.400102 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.402277 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.402285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.402292 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.402609 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.402944 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.402947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.402949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.403187 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.403659 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.403669 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.403925 (+ 256us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.404112 (+ 187us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.404475 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":22,"thread_start_us":159,"threads_started":1}
I20260809 01:52:55.405315 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.392595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.3.220.60:36917)
0809 01:52:55.393013 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.393027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.393062 (+ 35us) server_negotiation.cc:407] Connection header received
0809 01:52:55.393990 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.394013 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.394405 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.394652 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.395548 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.396430 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.397268 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.397706 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.397876 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.400766 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.400773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.400804 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.400835 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.402725 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.403294 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.403300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.403302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.403395 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.403708 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.403710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.403711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.403939 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.404113 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.404391 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.404530 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260809 01:52:55.406694 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.60@KRBTEST.COM for oryx/127.3.220.61@KRBTEST.COM
I20260809 01:52:55.414645 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.407137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.0.0.1:49602)
0809 01:52:55.407293 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.407309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.407380 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.407818 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.407820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.407939 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.408022 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.408027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.408911 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.408913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.409431 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.409438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.410024 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.410028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.410137 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.410562 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.410572 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.411726 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.413339 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.413342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.413343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.413515 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.413858 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.413860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.413861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.413933 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.414301 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.414303 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.414375 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.414453 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.414543 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 01:52:55.414747 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.407292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.3.220.61:41143)
0809 01:52:55.407602 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.407606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.407618 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:52:55.407661 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.407663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.407702 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.407788 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.408144 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.408816 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.409623 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.409884 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.410015 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.411862 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.411865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.411867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.411891 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.413244 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.413640 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.413643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.413644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.413765 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.414008 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.414010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.414011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.414217 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.414292 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.414509 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.414604 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":133,"threads_started":1}
I20260809 01:52:55.415594 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.60@KRBTEST.COM for oryx/127.3.220.60@KRBTEST.COM
I20260809 01:52:55.423488 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.416048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:36917 (local address 127.0.0.1:44340)
0809 01:52:55.416153 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.416168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.416242 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.416447 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.416449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.416567 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.416627 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.416631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.417749 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.417751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.418226 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.418233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.418634 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.418637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.418699 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.419249 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.419260 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.420390 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.422182 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.422185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.422186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.422361 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.422630 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.422632 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.422634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.422712 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.423104 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.423106 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.423178 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.423255 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.423348 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 01:52:55.423513 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.416202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.3.220.60:36917)
0809 01:52:55.416303 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.416306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.416316 (+ 10us) server_negotiation.cc:407] Connection header received
0809 01:52:55.416343 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.416345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.416376 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.416433 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.416717 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.417638 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.418320 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.418543 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.418616 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.420511 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.420514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.420516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.420537 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.422077 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.422466 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.422469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.422470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.422545 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.422834 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.422837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.422839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.423013 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.423103 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.423296 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.423396 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 01:52:55.426429 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Bootstrap starting.
I20260809 01:52:55.428712 13872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:55.429445 13872 log.cc:824] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:55.431501 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: No bootstrap required, opened a new log
I20260809 01:52:55.434682 13872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } }
I20260809 01:52:55.434906 13872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:55.434939 13872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419b6368ced541299d700c2059e48866, State: Initialized, Role: FOLLOWER
I20260809 01:52:55.435364 13872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } }
I20260809 01:52:55.435793 13879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } } }
I20260809 01:52:55.435930 13879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:52:55.436518 13872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 01:52:55.441241 13897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:52:55.441324 13897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:52:55.444815 13638 master_runner.cc:428] Detected that this master 419b6368ced541299d700c2059e48866 is joining an existing cluster
I20260809 01:52:55.444892 13638 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.220.60:36917
I20260809 01:52:55.456496 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.447436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.3.220.60:36917)
0809 01:52:55.447528 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.447532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.448298 (+ 766us) server_negotiation.cc:407] Connection header received
0809 01:52:55.448432 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.448435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.448468 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.448513 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.449103 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.449929 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.451284 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.451612 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.451697 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.453176 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.453180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.453181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.453206 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.454876 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.455341 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.455343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.455344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.455422 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.455719 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.455721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.455722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.455878 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.455952 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.456239 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.456377 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":37}
I20260809 01:52:55.456521 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.447499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:36917 (local address 127.0.0.1:44356)
0809 01:52:55.448227 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.448239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.448346 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.448630 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.448633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.448859 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.448987 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.448994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.450077 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.450080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.451163 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.451175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.451848 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.451852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.451941 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.452542 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.452557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.453039 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.454982 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.454985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.454987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.455240 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.455507 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.455510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.455512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.455610 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.455972 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.455975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.456079 (+ 104us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.456188 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.456325 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":35,"thread_start_us":113,"threads_started":1}
I20260809 01:52:55.458606 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 12914
I20260809 01:52:55.481040 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.447209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.0.0.1:49606)
0809 01:52:55.447854 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.447869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.447983 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.448368 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.448373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.448664 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.448914 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.448923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.450078 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.450081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.451171 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.451182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.451889 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.451895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.451984 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.476192 (+ 24208us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.476209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.476643 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.479110 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.479113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.479115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.479404 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.479754 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.479756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.479758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.479849 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.480387 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.480390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.480503 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.480605 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.480788 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":79,"spinlock_wait_cycles":13568,"thread_start_us":98,"threads_started":1}
I20260809 01:52:55.481338 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.446974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.3.220.62:45149)
0809 01:52:55.447101 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.447106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.447852 (+ 746us) server_negotiation.cc:407] Connection header received
0809 01:52:55.448041 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.448043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.448095 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.448167 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.449055 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.449942 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.451519 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.451861 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.451932 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.476015 (+ 24083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.476022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.476028 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.476072 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.478507 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.479971 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.479976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.479977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.480060 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.480392 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.480396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.480397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.480623 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.480689 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.481040 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.481174 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 01:52:55.481602 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.446964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.3.220.61:41143)
0809 01:52:55.447102 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.447106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.447955 (+ 849us) server_negotiation.cc:407] Connection header received
0809 01:52:55.448113 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.448118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.448172 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.448252 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.449055 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.449933 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.451314 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.451664 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.451733 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.476882 (+ 25149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.476886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.476891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.476930 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.478942 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.479522 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.479526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.479528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.479617 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.479965 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.479968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.479970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.480235 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.480311 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.481279 (+ 968us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.481416 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:52:55.481956 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.446911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:34634)
0809 01:52:55.447761 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.447773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.447891 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.448303 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.448306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.448664 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.448918 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.448923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.450078 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.450081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.451404 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.451412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.452129 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.452137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.452259 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.475300 (+ 23041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.475316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.475797 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.478661 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.478665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.478667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.479854 (+ 1187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.480178 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.480180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.480182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.480284 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.480765 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.480769 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.480854 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.481569 (+ 715us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.481737 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":98,"spinlock_wait_cycles":1920,"thread_start_us":103,"threads_started":1}
I20260809 01:52:55.483536 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:37357
--local_ip_for_outbound_sockets=127.3.220.1
--tserver_master_addrs=127.3.220.62:45149
--webserver_port=34437
--webserver_interface=127.3.220.1
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab
--principal=oryx/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143,127.3.220.60:36917 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240374, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.62@KRBTEST.COM for oryx/127.3.220.60@KRBTEST.COM
I20260809 01:52:55.497113 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.486740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:36917 (local address 127.0.0.1:44368)
0809 01:52:55.486864 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.486880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.486967 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.487299 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.487302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.487501 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.487612 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.487618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.488714 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.488718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.489693 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.489704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.490357 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.490361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.490465 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.491008 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.491023 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.492813 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.495053 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.495060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.495064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.495356 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.495735 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.495740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.495744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.495848 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.496411 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.496417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.496532 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.496670 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.496851 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 01:52:55.497781 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.486827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.3.220.60:36917)
0809 01:52:55.486941 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.486945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.486958 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:52:55.487095 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.487101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.487168 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.487251 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.487749 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.488581 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.489808 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.490223 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.490360 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.492993 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.493000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.493005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.493037 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.494907 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.495487 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.495493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.495496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.495586 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.495976 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.495981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.495983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.496249 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.496358 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.497359 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.497493 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 01:52:55.498781 12818 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.220.60:36917: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "419b6368ced541299d700c2059e48866"
member_type: NON_VOTER
last_known_addr {
host: "127.3.220.60"
port: 36917
}
attrs {
promote: true
}
}
dest_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef"
cas_config_opid_index: 5
I20260809 01:52:55.499472 12818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } }
I20260809 01:52:55.500559 13620 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 419b6368ced541299d700c2059e48866 (127.3.220.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.502152 13620 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.220.60:36917
I20260809 01:52:55.502324 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.502421 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:55.503471 13837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:55.505048 13837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 FOLLOWER]: Refusing update from remote peer 25c42ff3cd264b7082ea27fa4a0b42ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 01:52:55.505893 13619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:55.506912 13638 master.cc:537] Master@127.3.220.60:36917 shutting down...
I20260809 01:52:55.511188 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.502714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.0.0.1:49620)
0809 01:52:55.503023 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.503036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.503101 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.506267 (+ 3166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.506270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.506288 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:55.506372 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.506377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.507734 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.507738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.509351 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.509365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.510595 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.510598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.510801 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.510860 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.510964 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.511054 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260809 01:52:55.511373 13837 raft_consensus.cc:858] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } }
W20260809 01:52:55.511660 13837 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } }
I20260809 01:52:55.511919 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.502858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.3.220.61:41143)
0809 01:52:55.505821 (+ 2963us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.505826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.505840 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:55.505892 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.505895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.505957 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.506131 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:55.506503 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.507586 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.509500 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.510234 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.510347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.510518 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.511537 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.511615 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2871}
I20260809 01:52:55.512471 13713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 LEARNER]: Refusing update from remote peer 25c42ff3cd264b7082ea27fa4a0b42ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 01:52:55.512929 13619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 01:52:55.513157 13837 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8208ee9ce1d243faa7ee08a8c7177ada (127.3.220.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } }
I20260809 01:52:55.514725 13713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 419b6368ced541299d700c2059e48866 (127.3.220.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.515338 13879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c42ff3cd264b7082ea27fa4a0b42ef. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } }
I20260809 01:52:55.515435 13879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 01:52:55.515516 13879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } }
I20260809 01:52:55.515571 13879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 01:52:55.515626 13879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } }
I20260809 01:52:55.515689 13879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 01:52:55.516934 13755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c42ff3cd264b7082ea27fa4a0b42ef. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.517040 13755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: This master's current role is: LEARNER
I20260809 01:52:55.517125 13755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: NON_VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: true } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.517182 13755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: This master's current role is: LEARNER
I20260809 01:52:55.518209 13909 mvcc.cc:205] Tried to move back new op lower bound from 7316440572442304512 to 7316440572276834304. Current Snapshot: MvccSnapshot[applied={T|T < 7316440572309594112}]
I20260809 01:52:55.518854 13619 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: attempting to promote NON_VOTER 8208ee9ce1d243faa7ee08a8c7177ada to VOTER
I20260809 01:52:55.519304 13619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } }
I20260809 01:52:55.521045 13713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 LEARNER]: Refusing update from remote peer 25c42ff3cd264b7082ea27fa4a0b42ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260809 01:52:55.521132 12802 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef -> Peer 419b6368ced541299d700c2059e48866 (127.3.220.60:36917): Couldn't send request to peer 419b6368ced541299d700c2059e48866. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 01:52:55.521490 13620 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 01:52:55.523278 13620 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8208ee9ce1d243faa7ee08a8c7177ada (127.3.220.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.524370 13638 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260809 01:52:55.524607 13638 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260809 01:52:55.524677 13638 tablet_replica.cc:330] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: stopping tablet replica
I20260809 01:52:55.524657 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.524765 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:55.524881 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.524943 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:55.525729 13713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8208ee9ce1d243faa7ee08a8c7177ada (127.3.220.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.527195 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.527321 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:52:55.540827 13638 master.cc:552] Master@127.3.220.60:36917 shutdown complete.
I20260809 01:52:55.540901 13638 master_runner.cc:305] Clearing existing system tablet
I20260809 01:52:55.541180 13638 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:55.541262 13638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:55.542465 13638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:55.543325 13921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.543536 13638 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:55.543668 13638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:55.543718 13638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
uuid: "419b6368ced541299d700c2059e48866"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "f7e915ef276b71e6089663dbed0ba2ff"
server_key_iv: "7231b7ad69da06082bcf9789b45637cc"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.545360 13638 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:55.547600 13638 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 01:52:55.547705 13638 log.cc:1196] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 01:52:55.547960 13638 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Deleting consensus metadata
I20260809 01:52:55.548143 13638 master_runner.cc:315] Copying system tablet from 127.3.220.62:45149
I20260809 01:52:55.549258 13638 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: tablet copy: Beginning tablet copy session from remote peer at address 127.3.220.62:45149
I20260809 01:52:55.561920 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.549884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.3.220.62:45149)
0809 01:52:55.550016 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.550020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.553892 (+ 3872us) server_negotiation.cc:407] Connection header received
0809 01:52:55.554196 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.554199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.554258 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.554357 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.554949 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.555683 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.556678 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.556924 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.557047 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.558390 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.558394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.558397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.558426 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.559993 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.560529 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.560534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.560536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.560648 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.560966 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.560971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.560972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.561175 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.561298 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.561663 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.561773 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 01:52:55.562500 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.549810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:34636)
0809 01:52:55.553756 (+ 3946us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.553772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.554003 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.554395 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.554398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.554654 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.554813 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.554821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.555811 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.555817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.556529 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.556539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.557056 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.557060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.557195 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.557863 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.557876 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.558236 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.560128 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.560131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.560134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.560367 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.560756 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.560760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.560762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.560861 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.561287 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.561290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.561405 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.562189 (+ 784us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.562359 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3869,"thread_start_us":87,"threads_started":1}
I20260809 01:52:55.563167 12853 tablet_copy_service.cc:140] P 25c42ff3cd264b7082ea27fa4a0b42ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 419b6368ced541299d700c2059e48866 ({username='oryx', principal='oryx/127.3.220.60@KRBTEST.COM'} at 127.0.0.1:34636)
I20260809 01:52:55.563290 12853 tablet_copy_service.cc:161] P 25c42ff3cd264b7082ea27fa4a0b42ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 419b6368ced541299d700c2059e48866 at {username='oryx', principal='oryx/127.3.220.60@KRBTEST.COM'} at 127.0.0.1:34636: session id = 419b6368ced541299d700c2059e48866-00000000000000000000000000000000
I20260809 01:52:55.564307 12853 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 01:52:55.565654 13638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 01:52:55.567627 13638 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: tablet copy: Starting download of 0 data blocks...
I20260809 01:52:55.567865 13638 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: tablet copy: Starting download of 1 WAL segments...
I20260809 01:52:55.569376 13638 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 01:52:55.571135 12853 tablet_copy_service.cc:342] P 25c42ff3cd264b7082ea27fa4a0b42ef: Request end of tablet copy session 419b6368ced541299d700c2059e48866-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.220.60@KRBTEST.COM'} at 127.0.0.1:34636
I20260809 01:52:55.571247 12853 tablet_copy_service.cc:433] P 25c42ff3cd264b7082ea27fa4a0b42ef: ending tablet copy session 419b6368ced541299d700c2059e48866-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 419b6368ced541299d700c2059e48866
I20260809 01:52:55.578905 13638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:55.580771 13935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.581025 13940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.581230 13938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:55.581737 13638 server_base.cc:1034] running on GCE node
I20260809 01:52:55.581945 13638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:55.582315 13638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:55.583545 13638 hybrid_clock.cc:648] HybridClock initialized: now 1786240375583516 us; error 46 us; skew 500 ppm
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:55.585268 13638 init.cc:377] Logged in from keytab as oryx/127.3.220.60@KRBTEST.COM (short username oryx)
I20260809 01:52:55.585827 13638 webserver.cc:460] Webserver started at http://127.3.220.60:34127/ using document root <none> and password file <none>
I20260809 01:52:55.586069 13638 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:55.586141 13638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:55.587344 13638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:55.588403 13952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.588686 13638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 01:52:55.588796 13638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
uuid: "419b6368ced541299d700c2059e48866"
format_stamp: "Formatted at 2026-08-09 01:52:55 on dist-test-slave-gkw7"
server_key: "f7e915ef276b71e6089663dbed0ba2ff"
server_key_iv: "7231b7ad69da06082bcf9789b45637cc"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.588913 13638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:55.597707 13638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:55.599983 13638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:55.603030 13283 heartbeater.cc:503] Master 127.3.220.62:45149 was elected leader, sending a full tablet report...
I20260809 01:52:55.605744 13638 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.60:36917
I20260809 01:52:55.606082 14021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.60:36917 every 8 connection(s)
I20260809 01:52:55.606629 13638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
I20260809 01:52:55.608566 14024 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:52:55.609489 14024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Bootstrap starting.
I20260809 01:52:55.617218 14024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:52:55.617637 14024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: Bootstrap complete.
I20260809 01:52:55.618206 14024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } }
I20260809 01:52:55.618353 14024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 419b6368ced541299d700c2059e48866, State: Initialized, Role: LEARNER
I20260809 01:52:55.618484 14024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } }
I20260809 01:52:55.619019 14024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:52:55.619498 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:55.619717 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: LEARNER
I20260809 01:52:55.622679 14041 catalog_manager.cc:1259] Loaded cluster ID: b4288719cdb2436b861f8e28027d272e
I20260809 01:52:55.622793 14041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: loading cluster ID for follower catalog manager: success
I20260809 01:52:55.624886 14041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: acquiring CA information for follower catalog manager: success
I20260809 01:52:55.626221 14041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 01:52:55.648612 13908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:55.648948 13908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:55.649029 13908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:55.655424 13908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:55.655508 13908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:55.655596 13908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:52:55.662887 13908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab
--principal=oryx/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:37357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=34437
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143,127.3.220.60:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:55.664047 13908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:55.665133 13908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:55.674861 14049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.674870 14046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.674870 14047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:55.675169 13908 server_base.cc:1034] running on GCE node
I20260809 01:52:55.675557 13908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:55.676218 13908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:55.677412 13908 hybrid_clock.cc:648] HybridClock initialized: now 1786240375677397 us; error 27 us; skew 500 ppm
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:55.680279 13908 init.cc:377] Logged in from keytab as oryx/127.3.220.1@KRBTEST.COM (short username oryx)
I20260809 01:52:55.681197 13908 webserver.cc:460] Webserver started at http://127.3.220.1:34437/ using document root <none> and password file <none>
I20260809 01:52:55.681875 13908 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:55.681993 13908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:55.685920 13908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:52:55.688095 14058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.689456 13908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:55.689666 13908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "c782dcddc5764bd39e69380cc3ad640a"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "1deada40ec7280124496223136ad2f77"
server_key_iv: "769ee724d4745b2d4fec898ac41ec5ba"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.690290 13908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:55.713479 13908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:55.716856 13908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:55.717089 13908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:55.717823 13908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:55.718761 13908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:55.718839 13908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.718918 13908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:55.718966 13908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.729415 13908 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:37357
I20260809 01:52:55.729437 14229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:37357 every 8 connection(s)
I20260809 01:52:55.730543 13908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:52:55.735056 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 13908
I20260809 01:52:55.735304 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 13099
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.60@KRBTEST.COM
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.61@KRBTEST.COM
I20260809 01:52:55.750448 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.733132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:42763 (local address 127.3.220.62:45149)
0809 01:52:55.733272 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.733278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.734319 (+ 1041us) server_negotiation.cc:407] Connection header received
0809 01:52:55.735067 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.735070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.735149 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.735211 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.736946 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.737675 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.739171 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.739441 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.739509 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.744019 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.744024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.744027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.744058 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.746700 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.747707 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.747715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.747720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.747836 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.748328 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.748335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.748340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.748974 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.749080 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.749992 (+ 912us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.750160 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 01:52:55.753612 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.733105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:42763)
0809 01:52:55.734221 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.734235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.734798 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.735537 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.735546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.736178 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.736782 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.736798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.737822 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.737824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.739023 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.739033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.739701 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.739704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.739882 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.741031 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.741057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.743807 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.746878 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.746890 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.746915 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.747514 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.747966 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.747972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.747978 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.748188 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.749165 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.749175 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.749635 (+ 460us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.750744 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.752240 (+ 1496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":369,"thread_start_us":118,"threads_started":1}
I20260809 01:52:55.754912 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.732976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:58339 (local address 127.3.220.61:41143)
0809 01:52:55.733281 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.733287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.734260 (+ 973us) server_negotiation.cc:407] Connection header received
0809 01:52:55.735060 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.735064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.735126 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.735186 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.745219 (+ 10033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.746370 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.747484 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.747876 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.747967 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.751017 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.751024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.751027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.751056 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.752803 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.753493 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.753496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.753498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.753607 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.753946 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.753950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.753951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.754156 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.754286 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.754617 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.754738 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260809 01:52:55.755393 14231 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45149
I20260809 01:52:55.755568 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.733105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:36917 (local address 127.3.220.1:48163)
0809 01:52:55.733699 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.733758 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.734798 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.741939 (+ 7141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.741943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.742171 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.742331 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.742343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.743886 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.743893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.744720 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.744730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.745711 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.745715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.745843 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.748405 (+ 2562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.748424 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.750167 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.752716 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.752721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.752726 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.752973 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.753301 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.753305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.753307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.753387 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.754235 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.754241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.754539 (+ 298us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.754640 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.755247 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":132,"threads_started":1}
I20260809 01:52:55.755604 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.733105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.3.220.1:58339)
0809 01:52:55.734153 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.734166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.734803 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.735539 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.735548 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.744825 (+ 9277us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.745021 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.745036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.746518 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.746521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.747355 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.747367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.748158 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.748161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.748292 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.748819 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.748840 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.750837 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.753113 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.753116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.753118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.753366 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.753757 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.753761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.753765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.753850 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.754342 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.754344 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.754416 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.755056 (+ 640us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.755313 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":168,"spinlock_wait_cycles":6016,"thread_start_us":134,"threads_started":1}
I20260809 01:52:55.755685 14231 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:55.756318 14234 heartbeater.cc:347] Connected to a master server at 127.3.220.61:41143
I20260809 01:52:55.756361 14231 heartbeater.cc:511] Master 127.3.220.62:45149 requested a full tablet report, sending...
I20260809 01:52:55.756445 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.732741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:48163 (local address 127.3.220.60:36917)
0809 01:52:55.733460 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.733466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.734074 (+ 608us) server_negotiation.cc:407] Connection header received
0809 01:52:55.735154 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.735158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.735217 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.735302 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.742509 (+ 7207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.743723 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.744904 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.745383 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.745517 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.750339 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.750343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.750346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.750377 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.752566 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.753083 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.753089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.753091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.753186 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.753505 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.753508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.753510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.754093 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.754186 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.755437 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.756122 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":79,"threads_started":1}
I20260809 01:52:55.756408 14234 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:55.756644 14234 heartbeater.cc:511] Master 127.3.220.61:41143 requested a full tablet report, sending...
I20260809 01:52:55.757079 14230 heartbeater.cc:347] Connected to a master server at 127.3.220.60:36917
I20260809 01:52:55.757182 14230 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:55.757407 14230 heartbeater.cc:511] Master 127.3.220.60:36917 requested a full tablet report, sending...
I20260809 01:52:55.758109 13700 ts_manager.cc:194] Registered new tserver with Master: c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357)
I20260809 01:52:55.758498 13972 ts_manager.cc:194] Registered new tserver with Master: c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357)
I20260809 01:52:55.758569 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:43743
--local_ip_for_outbound_sockets=127.3.220.2
--tserver_master_addrs=127.3.220.62:45149
--webserver_port=38391
--webserver_interface=127.3.220.2
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab
--principal=oryx/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143,127.3.220.60:36917 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:52:55.758863 12818 ts_manager.cc:194] Re-registered known tserver with Master: c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357)
I20260809 01:52:55.759445 12818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:42763
I20260809 01:52:55.825250 13474 heartbeater.cc:503] Master 127.3.220.62:45149 was elected leader, sending a full tablet report...
W20260809 01:52:55.876772 14240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:55.877054 14240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:55.877130 14240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:55.881264 14240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:55.881357 14240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:55.881462 14240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:52:55.886674 14240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab
--principal=oryx/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:43743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=38391
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143,127.3.220.60:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:55.887777 14240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:55.888886 14240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:52:55.899554 14240 server_base.cc:1034] running on GCE node
W20260809 01:52:55.899500 14247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.899510 14249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:55.899493 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:55.900125 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:55.900763 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:55.901925 14240 hybrid_clock.cc:648] HybridClock initialized: now 1786240375901891 us; error 50 us; skew 500 ppm
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:55.904959 14240 init.cc:377] Logged in from keytab as oryx/127.3.220.2@KRBTEST.COM (short username oryx)
I20260809 01:52:55.905915 14240 webserver.cc:460] Webserver started at http://127.3.220.2:38391/ using document root <none> and password file <none>
I20260809 01:52:55.906490 14240 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:55.906566 14240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:55.910365 14240 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:55.912564 14256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.913940 14240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:55.914072 14240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "39f5e2bf2e0543a88e3d3d6783756314"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "b60af748fa870c80723bbf87685b1ace"
server_key_iv: "85c33c2c48baf443b48f81a123ab08c6"
server_key_version: "encryptionkey@0"
I20260809 01:52:55.914609 14240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:55.928450 14240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:55.931735 14240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:55.931959 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:55.932538 14240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:55.933389 14240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:55.933436 14240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.933475 14240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:55.933490 14240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:55.942672 14240 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:43743
I20260809 01:52:55.942698 14419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:43743 every 8 connection(s)
I20260809 01:52:55.943679 14240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:52:55.948633 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 14240
I20260809 01:52:55.948814 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 13290
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.61@KRBTEST.COM
I20260809 01:52:55.962188 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.945778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:44857 (local address 127.3.220.61:41143)
0809 01:52:55.945933 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.945938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.947176 (+ 1238us) server_negotiation.cc:407] Connection header received
0809 01:52:55.947929 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.947932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.947988 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.948042 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.949898 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.950757 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.952845 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.953189 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.953269 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.957758 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.957763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.957768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.957808 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.959637 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.960372 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.960378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.960381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.960463 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.960896 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.960901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.960903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.961110 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.961199 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.961869 (+ 670us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.962019 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 01:52:55.962273 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.945518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:54741 (local address 127.3.220.62:45149)
0809 01:52:55.945682 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.945686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.946799 (+ 1113us) server_negotiation.cc:407] Connection header received
0809 01:52:55.947929 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.947932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.947988 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.948043 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.950160 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.950916 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.952846 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.953199 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.953269 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.957299 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.957303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.957306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.957336 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.959723 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.960313 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.960316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.960318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.960419 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.960820 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.960828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.960829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.961083 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.961200 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.961874 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.962021 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 01:52:55.963997 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.946035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.3.220.2:44857)
0809 01:52:55.947077 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.947092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.947692 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.948248 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.948256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.948980 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.949722 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.949742 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.950976 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.950980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.952703 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.952714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.953480 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.953484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.953828 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.954840 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.954859 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.957569 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.959780 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.959789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.959808 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.960271 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.960577 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.960583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.960587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.960725 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.961248 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.961251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.961519 (+ 268us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.962749 (+ 1230us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.963024 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":462,"spinlock_wait_cycles":30976,"thread_start_us":156,"threads_started":1}
I20260809 01:52:55.963999 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.946034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:54741)
0809 01:52:55.946641 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.946693 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.947692 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.948248 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.948256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.949887 (+ 1631us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.949996 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.950002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.951079 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.951082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.952703 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.952714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.953480 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.953484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.953832 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.954746 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.954777 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.957110 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.959808 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.959811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.959813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.960199 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.960535 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.960540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.960542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.960707 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.961224 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.961233 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.961551 (+ 318us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.962514 (+ 963us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.962922 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":232,"spinlock_wait_cycles":16896,"thread_start_us":159,"threads_started":1}
I20260809 01:52:55.965687 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:36817
--local_ip_for_outbound_sockets=127.3.220.3
--tserver_master_addrs=127.3.220.62:45149
--webserver_port=38941
--webserver_interface=127.3.220.3
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab
--principal=oryx/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143,127.3.220.60:36917 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:52:55.965795 14423 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45149
I20260809 01:52:55.965801 14425 heartbeater.cc:347] Connected to a master server at 127.3.220.61:41143
I20260809 01:52:55.966056 14425 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:55.966056 14423 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:55.966730 14425 heartbeater.cc:511] Master 127.3.220.61:41143 requested a full tablet report, sending...
I20260809 01:52:55.966732 14423 heartbeater.cc:511] Master 127.3.220.62:45149 requested a full tablet report, sending...
I20260809 01:52:55.967864 12818 ts_manager.cc:194] Re-registered known tserver with Master: 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743)
I20260809 01:52:55.968011 13700 ts_manager.cc:194] Registered new tserver with Master: 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743)
Aug 09 01:52:55 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240375, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.60@KRBTEST.COM
I20260809 01:52:55.968698 12818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:54741
I20260809 01:52:55.972710 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.946261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:51659 (local address 127.3.220.60:36917)
0809 01:52:55.946400 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:55.946404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:55.947236 (+ 832us) server_negotiation.cc:407] Connection header received
0809 01:52:55.947905 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:55.947908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:55.947969 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:55.948077 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:55.962089 (+ 14012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.962935 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.964094 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.964494 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.964586 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.969155 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:55.969196 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:55.969199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:55.969233 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:55.970931 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.971450 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.971452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.971454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.971527 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:55.971885 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:55.971888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:55.971889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:55.972053 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:55.972192 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:55.972446 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 01:52:55.972576 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 01:52:55.972729 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:55.946035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:36917 (local address 127.3.220.2:51659)
0809 01:52:55.947147 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:55.947162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:55.947702 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:55.948247 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:55.948254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:55.949331 (+ 1077us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:55.949722 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.949742 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.963148 (+ 13406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.963151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.963941 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:55.963951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:55.964785 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:55.964790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:55.964893 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:55.967173 (+ 2280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:55.967186 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:55.968847 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:55.971050 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.971056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.971060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.971349 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.971627 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:55.971633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:55.971638 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:55.971781 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:55.972159 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:55.972162 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:55.972264 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 01:52:55.972380 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 01:52:55.972536 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":167,"thread_start_us":118,"threads_started":1}
I20260809 01:52:55.973500 14422 heartbeater.cc:347] Connected to a master server at 127.3.220.60:36917
I20260809 01:52:55.973665 14422 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:55.973915 14422 heartbeater.cc:511] Master 127.3.220.60:36917 requested a full tablet report, sending...
I20260809 01:52:55.974726 13972 ts_manager.cc:194] Registered new tserver with Master: 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743)
I20260809 01:52:56.056795 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.052629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.3.220.60:36917)
0809 01:52:56.052755 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.052759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.052838 (+ 79us) server_negotiation.cc:407] Connection header received
0809 01:52:56.053054 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.053056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.053111 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.053219 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:56.053603 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.054258 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.055826 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.056248 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.056329 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.056446 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.056618 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.056688 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 01:52:56.056847 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.052421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:36917 (local address 127.0.0.1:44380)
0809 01:52:56.052741 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.052762 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.052908 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.053268 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.053271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.053293 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:56.053435 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.053442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.054391 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.054395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.055695 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.055706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.056376 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.056380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.056482 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.056510 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.056596 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.056690 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20260809 01:52:56.057576 13975 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 01:52:56.057865 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c42ff3cd264b7082ea27fa4a0b42ef. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:56.058036 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: LEARNER
W20260809 01:52:56.088088 14433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:52:56.088362 14433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:52:56.088406 14433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:52:56.092669 14433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:52:56.092725 14433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:52:56.092801 14433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:52:56.098497 14433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab
--principal=oryx/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:36817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=38941
--tserver_master_addrs=127.3.220.62:45149,127.3.220.61:41143,127.3.220.60:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:52:56.099571 14433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:52:56.100651 14433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:52:56.111116 14442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:52:56.111136 14441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:56.111436 14433 server_base.cc:1034] running on GCE node
W20260809 01:52:56.111529 14446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:52:56.111968 14433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:52:56.112617 14433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:52:56.113848 14433 hybrid_clock.cc:648] HybridClock initialized: now 1786240376113811 us; error 45 us; skew 500 ppm
Aug 09 01:52:56 dist-test-slave-gkw7 krb5kdc[12764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240376, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:52:56.116575 14433 init.cc:377] Logged in from keytab as oryx/127.3.220.3@KRBTEST.COM (short username oryx)
I20260809 01:52:56.117512 14433 webserver.cc:460] Webserver started at http://127.3.220.3:38941/ using document root <none> and password file <none>
I20260809 01:52:56.118160 14433 fs_manager.cc:359] Metadata directory not provided
I20260809 01:52:56.118271 14433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:52:56.122527 14433 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 01:52:56.124567 14456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:56.125936 14433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:52:56.126080 14433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "fd69de1d46fe4e24a3726b915c893e05"
format_stamp: "Formatted at 2026-08-09 01:52:54 on dist-test-slave-gkw7"
server_key: "6c342fd903c2a35ba1ae93668246109c"
server_key_iv: "00bd7acbb0ad8471dc276eea9a5ec467"
server_key_version: "encryptionkey@0"
I20260809 01:52:56.126626 14433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:52:56.146152 14433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:52:56.149178 14433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:52:56.149387 14433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:52:56.150081 14433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:52:56.151057 14433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:52:56.151139 14433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:56.151196 14433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:52:56.151242 14433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:52:56.160523 14433 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:36817
I20260809 01:52:56.160547 14627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:36817 every 8 connection(s)
I20260809 01:52:56.161643 14433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:52:56.165462 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 14433
Aug 09 01:52:56 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240376, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
Aug 09 01:52:56 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240376, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.60@KRBTEST.COM
Aug 09 01:52:56 dist-test-slave-gkw7 krb5kdc[12764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240376, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.61@KRBTEST.COM
I20260809 01:52:56.182297 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.167583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.3.220.62:45149)
0809 01:52:56.168918 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.168923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.168937 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:56.168971 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.168976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.169027 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.169123 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.170116 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.170916 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.172138 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.172490 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.172585 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.175506 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.175510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.175513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.175539 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.180133 (+ 4594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.180725 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.180732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.180736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.180859 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.181186 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.181189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.181191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.181419 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.181519 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.181976 (+ 457us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.182115 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"spinlock_wait_cycles":384,"thread_start_us":83,"threads_started":1}
I20260809 01:52:56.182368 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.167865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.3.220.60:36917)
0809 01:52:56.168142 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.168146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.169445 (+ 1299us) server_negotiation.cc:407] Connection header received
0809 01:52:56.169655 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.169658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.169708 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.169773 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.170716 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.171456 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.173211 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.173622 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.173713 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.177492 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.177496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.177499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.177556 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.179918 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.180562 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.180566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.180568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.180661 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.180994 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.181001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.181005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.181273 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.181395 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.182014 (+ 619us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.182157 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":102,"threads_started":1}
I20260809 01:52:56.182796 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.163864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:37343 (local address 127.3.220.62:45149)
0809 01:52:56.164143 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.164148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.165073 (+ 925us) server_negotiation.cc:407] Connection header received
0809 01:52:56.166115 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.166119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.166178 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.166233 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.168133 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.168919 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.170834 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.171163 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.171236 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.175821 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.175826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.175828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.175861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.179723 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.180809 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.180812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.180814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.180884 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.181318 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.181321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.181322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.181487 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.181597 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.182546 (+ 949us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.182650 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":293,"server-negotiator.queue_time_us":196}
I20260809 01:52:56.184852 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.164242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:37343)
0809 01:52:56.164977 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.164991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.165884 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.166624 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.166632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.167253 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:56.167959 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.167977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.169102 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.169108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.170695 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.170706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.171426 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.171429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.171657 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.172909 (+ 1252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:56.172930 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:56.175618 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:56.180281 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:56.180291 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:56.180326 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:56.180705 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:56.181002 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:56.181005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:56.181007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:56.181200 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:56.181669 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:56.181890 (+ 221us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:56.182221 (+ 331us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.183383 (+ 1162us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.183751 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":63,"spinlock_wait_cycles":15616,"thread_start_us":153,"threads_started":1}
I20260809 01:52:56.185253 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.164242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.3.220.3:35913)
0809 01:52:56.165816 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.165830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.165974 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.166622 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.166631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.167239 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:56.168210 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.168221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.169241 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.169246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.172124 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.172134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.173271 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.173274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.173377 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.174680 (+ 1303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:56.174695 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:56.177303 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:56.181066 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:56.181069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:56.181071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:56.181296 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:56.182157 (+ 861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:56.182159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:56.182161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:56.182247 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:56.184355 (+ 2108us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:56.184357 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:56.184456 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.184558 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.184730 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"mutex_wait_us":2882,"spinlock_wait_cycles":29568,"thread_start_us":169,"threads_started":1}
I20260809 01:52:56.186054 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.164001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:35913 (local address 127.3.220.61:41143)
0809 01:52:56.164262 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.164266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.165911 (+ 1645us) server_negotiation.cc:407] Connection header received
0809 01:52:56.166091 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.166095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.166149 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.166210 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.168351 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.169073 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.172295 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.172644 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.172747 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.178298 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.178315 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.178319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.178348 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.180582 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.181943 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.181948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.181950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.182040 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.184001 (+ 1961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.184007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.184009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.184224 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.185698 (+ 1474us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.185780 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.185908 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260809 01:52:56.186112 14629 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45149
I20260809 01:52:56.186383 14629 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:56.186455 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.163886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:39581 (local address 127.3.220.60:36917)
0809 01:52:56.164042 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.164047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.164944 (+ 897us) server_negotiation.cc:407] Connection header received
0809 01:52:56.166097 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.166100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.166149 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.166215 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.169459 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.170279 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.172336 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.172659 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.172742 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.178043 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.178049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.178053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.178083 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.180239 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.182356 (+ 2117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.182359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.182361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.182472 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.185494 (+ 3022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.185497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.185499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.185757 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.185843 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.186183 (+ 340us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.186305 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 01:52:56.186766 14630 heartbeater.cc:347] Connected to a master server at 127.3.220.61:41143
I20260809 01:52:56.186842 14630 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:56.186973 14630 heartbeater.cc:511] Master 127.3.220.61:41143 requested a full tablet report, sending...
I20260809 01:52:56.186959 14629 heartbeater.cc:511] Master 127.3.220.62:45149 requested a full tablet report, sending...
I20260809 01:52:56.188187 13700 ts_manager.cc:194] Registered new tserver with Master: fd69de1d46fe4e24a3726b915c893e05 (127.3.220.3:36817)
I20260809 01:52:56.188419 12823 ts_manager.cc:194] Re-registered known tserver with Master: fd69de1d46fe4e24a3726b915c893e05 (127.3.220.3:36817)
I20260809 01:52:56.189209 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.168295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.3.220.61:41143)
0809 01:52:56.173878 (+ 5583us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.173882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.173894 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:52:56.173927 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.173930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.173974 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.174058 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.175324 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.176116 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.177236 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.177649 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.177763 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.179753 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.179759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.179765 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.179795 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.186701 (+ 6906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.187150 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.187153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.187155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.187239 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.187549 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.187552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.187553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.187747 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.187882 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.188928 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.189046 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5500,"thread_start_us":85,"threads_started":1}
I20260809 01:52:56.189283 12823 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:37343
I20260809 01:52:56.186442 12818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:52:56.190289 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.164242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:36917 (local address 127.3.220.3:39581)
0809 01:52:56.164788 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.164839 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.165867 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.166622 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.166631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.167453 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:52:56.167956 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.167973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.170413 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.170416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.171237 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.171247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.172899 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.172902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.173018 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.173655 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:52:56.173683 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:52:56.176900 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:52:56.180712 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:56.180717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:56.180722 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:56.180978 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:56.182578 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:52:56.182581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:52:56.182583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:52:56.182688 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:52:56.185899 (+ 3211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:52:56.185901 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:52:56.185988 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.188092 (+ 2104us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.190120 (+ 2028us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":7,"spinlock_wait_cycles":8832,"thread_start_us":138,"threads_started":1}
W20260809 01:52:56.190846 12818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:56.191672 14628 heartbeater.cc:347] Connected to a master server at 127.3.220.60:36917
I20260809 01:52:56.191759 14628 heartbeater.cc:464] Registering TS with master...
I20260809 01:52:56.191951 14628 heartbeater.cc:511] Master 127.3.220.60:36917 requested a full tablet report, sending...
I20260809 01:52:56.192629 13972 ts_manager.cc:194] Registered new tserver with Master: fd69de1d46fe4e24a3726b915c893e05 (127.3.220.3:36817)
I20260809 01:52:56.193617 14434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: NON_VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: true } } }
I20260809 01:52:56.193723 14434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:56.195842 14651 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: attempting to promote NON_VOTER 419b6368ced541299d700c2059e48866 to VOTER
I20260809 01:52:56.196185 14651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } }
I20260809 01:52:56.197201 13713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Refusing update from remote peer 25c42ff3cd264b7082ea27fa4a0b42ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 01:52:56.197202 13975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 LEARNER]: Refusing update from remote peer 25c42ff3cd264b7082ea27fa4a0b42ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 01:52:56.197695 14434 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 01:52:56.197831 14649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 01:52:56.199666 14651 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 419b6368ced541299d700c2059e48866 (127.3.220.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:56.200076 13713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 419b6368ced541299d700c2059e48866 (127.3.220.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:56.200076 13975 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 419b6368ced541299d700c2059e48866 (127.3.220.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:56.201247 14434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:56.201385 14434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:56.201484 14649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:56.201609 14649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:56.201272 14651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:56.201982 14651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:56.202152 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:56.202262 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:52:56.205981 14665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:56.206118 14665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:52:56.213994 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.207811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:37357 (local address 127.0.0.1:49614)
0809 01:52:56.207948 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.207968 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.208059 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.209022 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.209026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.209047 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:56.209174 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.209182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.211078 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.211094 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.212276 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.212289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.213569 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.213573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.213668 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.213695 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.213760 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.213833 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260809 01:52:56.214540 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.208053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.3.220.1:37357)
0809 01:52:56.208417 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.208439 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.208468 (+ 29us) server_negotiation.cc:407] Connection header received
0809 01:52:56.208562 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.208568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.208827 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.209358 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:56.209534 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.210956 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.212464 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.213400 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.214000 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.214196 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.214260 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.214368 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":150,"threads_started":1}
I20260809 01:52:56.216202 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.208324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:36817 (local address 127.0.0.1:39066)
0809 01:52:56.211215 (+ 2891us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.211231 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.211340 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.211933 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.211936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.211989 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:56.212076 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.212081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.213405 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.213410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.214862 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.214873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.215765 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.215769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.215861 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.215890 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.215983 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.216054 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2821,"thread_start_us":103,"threads_started":1}
I20260809 01:52:56.216248 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.208769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.3.220.3:36817)
0809 01:52:56.209112 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.209127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.211363 (+ 2236us) server_negotiation.cc:407] Connection header received
0809 01:52:56.211450 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.211454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.211620 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.211837 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:56.212234 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.213294 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.214993 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.215653 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.215767 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.215899 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.216009 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.216099 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
I20260809 01:52:56.217324 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.208566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.3.220.2:43743)
0809 01:52:56.211995 (+ 3429us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.212008 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.212033 (+ 25us) server_negotiation.cc:407] Connection header received
0809 01:52:56.212112 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.212117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.212301 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.212466 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:56.212663 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.213763 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.215646 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.216496 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.216615 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.216765 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.217063 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.217163 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3316,"thread_start_us":117,"threads_started":1}
I20260809 01:52:56.217943 14136 tablet_service.cc:1467] Processing CreateTablet for tablet 559d40898bd0430cb3cf5dc635f65309 (DEFAULT_TABLE table=test-table [id=7fb8730068ab4d3191ffd5a7212c584e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:56.219120 14136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559d40898bd0430cb3cf5dc635f65309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:56.219476 14523 tablet_service.cc:1467] Processing CreateTablet for tablet 559d40898bd0430cb3cf5dc635f65309 (DEFAULT_TABLE table=test-table [id=7fb8730068ab4d3191ffd5a7212c584e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:56.220587 14523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559d40898bd0430cb3cf5dc635f65309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:56.221721 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.208324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:43743 (local address 127.0.0.1:36068)
0809 01:52:56.211531 (+ 3207us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.211546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.211643 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.212453 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.212456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.212476 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:56.212561 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.212566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.213906 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.213909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.215476 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.215489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.216827 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.216830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.216897 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.216916 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.217028 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.217114 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3150,"mutex_wait_us":163,"thread_start_us":93,"threads_started":1}
I20260809 01:52:56.224836 14328 tablet_service.cc:1467] Processing CreateTablet for tablet 559d40898bd0430cb3cf5dc635f65309 (DEFAULT_TABLE table=test-table [id=7fb8730068ab4d3191ffd5a7212c584e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:56.225920 14328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559d40898bd0430cb3cf5dc635f65309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:56.227564 14676 tablet_bootstrap.cc:492] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: Bootstrap starting.
I20260809 01:52:56.230219 14676 tablet_bootstrap.cc:654] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:56.231017 14676 log.cc:824] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:56.231823 14679 tablet_bootstrap.cc:492] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: Bootstrap starting.
I20260809 01:52:56.233276 14676 tablet_bootstrap.cc:492] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: No bootstrap required, opened a new log
I20260809 01:52:56.233601 14676 ts_tablet_manager.cc:1397] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:52:56.233955 14679 tablet_bootstrap.cc:654] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:56.235119 14679 log.cc:824] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:56.237159 14679 tablet_bootstrap.cc:492] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: No bootstrap required, opened a new log
I20260809 01:52:56.237340 14679 ts_tablet_manager.cc:1397] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:52:56.237643 14676 raft_consensus.cc:348] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.237933 14676 raft_consensus.cc:374] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:56.237957 14681 tablet_bootstrap.cc:492] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: Bootstrap starting.
I20260809 01:52:56.237995 14676 raft_consensus.cc:729] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c782dcddc5764bd39e69380cc3ad640a, State: Initialized, Role: FOLLOWER
I20260809 01:52:56.238504 14676 consensus_queue.cc:260] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.239552 14676 ts_tablet_manager.cc:1428] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:52:56.240301 14681 tablet_bootstrap.cc:654] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:56.241169 14681 log.cc:824] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: Log is configured to *not* fsync() on all Append() calls
I20260809 01:52:56.242182 14679 raft_consensus.cc:348] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.242388 14231 heartbeater.cc:503] Master 127.3.220.62:45149 was elected leader, sending a full tablet report...
I20260809 01:52:56.242470 14679 raft_consensus.cc:374] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:56.242540 14679 raft_consensus.cc:729] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd69de1d46fe4e24a3726b915c893e05, State: Initialized, Role: FOLLOWER
I20260809 01:52:56.243064 14679 consensus_queue.cc:260] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.244055 14679 ts_tablet_manager.cc:1428] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 01:52:56.246374 14629 heartbeater.cc:503] Master 127.3.220.62:45149 was elected leader, sending a full tablet report...
I20260809 01:52:56.250247 14681 tablet_bootstrap.cc:492] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: No bootstrap required, opened a new log
I20260809 01:52:56.250474 14681 ts_tablet_manager.cc:1397] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20260809 01:52:56.254281 14681 raft_consensus.cc:348] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.254501 14681 raft_consensus.cc:374] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:56.254535 14681 raft_consensus.cc:729] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f5e2bf2e0543a88e3d3d6783756314, State: Initialized, Role: FOLLOWER
I20260809 01:52:56.255046 14681 consensus_queue.cc:260] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.256305 14681 ts_tablet_manager.cc:1428] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:52:56.256780 14423 heartbeater.cc:503] Master 127.3.220.62:45149 was elected leader, sending a full tablet report...
I20260809 01:52:56.273947 14686 raft_consensus.cc:482] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:56.274170 14686 raft_consensus.cc:504] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.275394 14686 leader_election.cc:288] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357), 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743)
I20260809 01:52:56.279836 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.275828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:35275 (local address 127.3.220.2:43743)
0809 01:52:56.275940 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.275947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.275960 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:52:56.276059 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.276061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.276108 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.276202 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:56.276703 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.277501 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.278798 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.279232 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.279357 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.279422 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.279639 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.279708 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 01:52:56.279838 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.275678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:43743 (local address 127.3.220.3:35275)
0809 01:52:56.275867 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.275882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.275951 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.276256 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.276259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.276287 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:56.276573 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.276580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.277681 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.277683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.278664 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.278676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.279358 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.279362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.279504 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.279521 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.279605 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.279677 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":139,"spinlock_wait_cycles":21248}
I20260809 01:52:56.279851 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.275735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:39811 (local address 127.3.220.1:37357)
0809 01:52:56.275925 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.275930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.275951 (+ 21us) server_negotiation.cc:407] Connection header received
0809 01:52:56.276051 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.276054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.276115 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.276230 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:56.276703 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.277523 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.278805 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.279232 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.279358 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.279426 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.279656 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.279728 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 01:52:56.279855 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.275603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:37357 (local address 127.3.220.3:39811)
0809 01:52:56.275836 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:56.275852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:56.275951 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:56.276256 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:56.276259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:56.276289 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:56.276576 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.276581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.277674 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.277677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.278658 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:56.278668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.279360 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.279364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:56.279532 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.279550 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 01:52:56.279638 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 01:52:56.279713 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":185,"spinlock_wait_cycles":7040}
I20260809 01:52:56.280537 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559d40898bd0430cb3cf5dc635f65309" candidate_uuid: "fd69de1d46fe4e24a3726b915c893e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f5e2bf2e0543a88e3d3d6783756314" is_pre_election: true
I20260809 01:52:56.280532 14164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559d40898bd0430cb3cf5dc635f65309" candidate_uuid: "fd69de1d46fe4e24a3726b915c893e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c782dcddc5764bd39e69380cc3ad640a" is_pre_election: true
I20260809 01:52:56.280831 14359 raft_consensus.cc:2454] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd69de1d46fe4e24a3726b915c893e05 in term 0.
I20260809 01:52:56.280864 14164 raft_consensus.cc:2454] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd69de1d46fe4e24a3726b915c893e05 in term 0.
I20260809 01:52:56.281332 14461 leader_election.cc:302] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f5e2bf2e0543a88e3d3d6783756314, fd69de1d46fe4e24a3726b915c893e05; no voters:
I20260809 01:52:56.281644 14686 raft_consensus.cc:2781] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:56.281750 14686 raft_consensus.cc:482] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:56.281818 14686 raft_consensus.cc:3037] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:56.282912 14686 raft_consensus.cc:504] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.283288 14686 leader_election.cc:288] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [CANDIDATE]: Term 1 election: Requested vote from peers c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357), 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743)
I20260809 01:52:56.283653 14164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559d40898bd0430cb3cf5dc635f65309" candidate_uuid: "fd69de1d46fe4e24a3726b915c893e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c782dcddc5764bd39e69380cc3ad640a"
I20260809 01:52:56.283679 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559d40898bd0430cb3cf5dc635f65309" candidate_uuid: "fd69de1d46fe4e24a3726b915c893e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f5e2bf2e0543a88e3d3d6783756314"
I20260809 01:52:56.283797 14164 raft_consensus.cc:3037] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:56.283818 14359 raft_consensus.cc:3037] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:56.285017 14164 raft_consensus.cc:2454] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd69de1d46fe4e24a3726b915c893e05 in term 1.
I20260809 01:52:56.285110 14359 raft_consensus.cc:2454] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd69de1d46fe4e24a3726b915c893e05 in term 1.
I20260809 01:52:56.285367 14460 leader_election.cc:302] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c782dcddc5764bd39e69380cc3ad640a, fd69de1d46fe4e24a3726b915c893e05; no voters:
I20260809 01:52:56.285614 14686 raft_consensus.cc:2781] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:56.285863 14686 raft_consensus.cc:686] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 1 LEADER]: Becoming Leader. State: Replica: fd69de1d46fe4e24a3726b915c893e05, State: Running, Role: LEADER
I20260809 01:52:56.286248 14686 consensus_queue.cc:237] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } }
I20260809 01:52:56.289779 12823 catalog_manager.cc:5495] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd69de1d46fe4e24a3726b915c893e05 (127.3.220.3). New cstate: current_term: 1 leader_uuid: "fd69de1d46fe4e24a3726b915c893e05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } health_report { overall_health: HEALTHY } } }
I20260809 01:52:56.314143 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.310079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.3.220.3:36817)
0809 01:52:56.310222 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.310227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.310243 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:52:56.310419 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.310422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.310494 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.310609 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:56.310865 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.311406 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.312400 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.312642 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.312795 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.313124 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:56.313654 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:56.313794 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.313866 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.313929 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 01:52:56.320724 14164 raft_consensus.cc:1261] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 1 FOLLOWER]: Refusing update from remote peer fd69de1d46fe4e24a3726b915c893e05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:56.320749 14359 raft_consensus.cc:1261] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 1 FOLLOWER]: Refusing update from remote peer fd69de1d46fe4e24a3726b915c893e05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:56.321508 14691 consensus_queue.cc:1035] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:56.321777 14686 consensus_queue.cc:1035] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:56.332017 14697 mvcc.cc:205] Tried to move back new op lower bound from 7316440581402140672 to 7316440581271298048. Current Snapshot: MvccSnapshot[applied={T|T < 7316440581402140672}]
I20260809 01:52:56.333552 14698 mvcc.cc:205] Tried to move back new op lower bound from 7316440581402140672 to 7316440581271298048. Current Snapshot: MvccSnapshot[applied={T|T < 7316440581402140672}]
I20260809 01:52:56.342413 12823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:56.342597 12823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:56.346705 12823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25c42ff3cd264b7082ea27fa4a0b42ef: Sending DeleteTablet for 3 replicas of tablet 559d40898bd0430cb3cf5dc635f65309
I20260809 01:52:56.347848 14328 tablet_service.cc:1514] Processing DeleteTablet for tablet 559d40898bd0430cb3cf5dc635f65309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:56 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:36068
I20260809 01:52:56.347952 14136 tablet_service.cc:1514] Processing DeleteTablet for tablet 559d40898bd0430cb3cf5dc635f65309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:56 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:49614
I20260809 01:52:56.348425 14713 tablet_replica.cc:330] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: stopping tablet replica
I20260809 01:52:56.348582 14523 tablet_service.cc:1514] Processing DeleteTablet for tablet 559d40898bd0430cb3cf5dc635f65309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:56 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:39066
I20260809 01:52:56.348773 14713 raft_consensus.cc:2229] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:56.349109 14713 raft_consensus.cc:2258] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:56.349512 14712 tablet_replica.cc:330] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: stopping tablet replica
I20260809 01:52:56.349920 14712 raft_consensus.cc:2229] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:56.350134 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 12780
I20260809 01:52:56.356950 14712 raft_consensus.cc:2258] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:56.358338 14713 ts_tablet_manager.cc:1905] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:56.358691 14712 ts_tablet_manager.cc:1905] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:56.361238 14713 ts_tablet_manager.cc:1918] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:56.361335 14713 log.cc:1196] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/559d40898bd0430cb3cf5dc635f65309
I20260809 01:52:56.361711 14713 ts_tablet_manager.cc:1939] T 559d40898bd0430cb3cf5dc635f65309 P c782dcddc5764bd39e69380cc3ad640a: Deleting consensus metadata
I20260809 01:52:56.362089 14712 ts_tablet_manager.cc:1918] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:56.362205 14712 log.cc:1196] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/559d40898bd0430cb3cf5dc635f65309
I20260809 01:52:56.362587 14712 ts_tablet_manager.cc:1939] T 559d40898bd0430cb3cf5dc635f65309 P 39f5e2bf2e0543a88e3d3d6783756314: Deleting consensus metadata
I20260809 01:52:56.363160 14716 tablet_replica.cc:330] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: stopping tablet replica
I20260809 01:52:56.363492 14716 raft_consensus.cc:2229] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:52:56.363931 14716 raft_consensus.cc:2258] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:56.365480 14716 ts_tablet_manager.cc:1905] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 01:52:56.366333 14259 connection.cc:535] server connection from 127.0.0.1:36068 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 01:52:56.366611 14060 connection.cc:535] server connection from 127.0.0.1:49614 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 01:52:56.368992 14716 ts_tablet_manager.cc:1918] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:56.369087 14716 log.cc:1196] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/559d40898bd0430cb3cf5dc635f65309
I20260809 01:52:56.369415 14716 ts_tablet_manager.cc:1939] T 559d40898bd0430cb3cf5dc635f65309 P fd69de1d46fe4e24a3726b915c893e05: Deleting consensus metadata
W20260809 01:52:56.370595 14459 connection.cc:441] server connection from 127.0.0.1:39066 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39066 (request call id 1) could send its response
I20260809 01:52:56.377785 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.349014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.3.220.61:41143)
0809 01:52:56.349159 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.349164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.367145 (+ 17981us) server_negotiation.cc:407] Connection header received
0809 01:52:56.367401 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.367405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.367485 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.367567 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.368635 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.369404 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.370581 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.370942 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.371035 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.373119 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.373123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.373126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.373152 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.374969 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.375815 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.375818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.375820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.375962 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.376564 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.376568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.376570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.376842 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.376955 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.377358 (+ 403us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.377511 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 01:52:56.380244 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.369726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.3.220.61:41143)
0809 01:52:56.369852 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.369856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.371028 (+ 1172us) server_negotiation.cc:407] Connection header received
0809 01:52:56.371189 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.371192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.371236 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.371285 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.372285 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.373054 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.374307 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.374649 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.374733 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.376406 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.376410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.376413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.376438 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.378189 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.378843 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.378846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.378848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.378937 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.379271 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.379275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.379277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.379478 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.379577 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.379923 (+ 346us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.380042 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 01:52:56.382448 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.370379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.3.220.60:36917)
0809 01:52:56.370500 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.370507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.371517 (+ 1010us) server_negotiation.cc:407] Connection header received
0809 01:52:56.371701 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.371707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.371774 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.371829 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.372872 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.373729 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.375090 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.375498 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.375590 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.377216 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.377220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.377225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.377260 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.379901 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.380871 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.380875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.380877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.380981 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.381342 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.381346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.381348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.381595 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.381720 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.382116 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.382281 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 01:52:56.389433 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:56.349034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.3.220.60:36917)
0809 01:52:56.349246 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:56.349251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:56.379623 (+ 30372us) server_negotiation.cc:407] Connection header received
0809 01:52:56.379832 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:56.379837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:56.379892 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:56.379954 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:52:56.381187 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.382081 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.383345 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:56.383690 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:56.383796 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:56.385417 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:52:56.385421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:52:56.385423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:52:56.385447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:52:56.387661 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.388119 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.388122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.388123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.388205 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:52:56.388503 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:52:56.388505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:52:56.388506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:52:56.388745 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:52:56.388894 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:56.389130 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 01:52:56.389253 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260809 01:52:57.334609 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.334076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:45347)
0809 01:52:57.334375 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.334489 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":129,"threads_started":1}
W20260809 01:52:57.334983 14231 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:45149 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:57.337143 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.336689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:50429)
0809 01:52:57.336931 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.337035 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":101,"threads_started":1}
W20260809 01:52:57.337599 14423 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:45149 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:57.337644 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.337165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:39921)
0809 01:52:57.337401 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.337512 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
W20260809 01:52:57.337939 14629 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:45149 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:57.846657 14741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25c42ff3cd264b7082ea27fa4a0b42ef)
I20260809 01:52:57.846897 14741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } }
I20260809 01:52:57.847314 14742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25c42ff3cd264b7082ea27fa4a0b42ef)
I20260809 01:52:57.847483 14742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } }
I20260809 01:52:57.848045 14741 leader_election.cc:288] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149), 8208ee9ce1d243faa7ee08a8c7177ada (127.3.220.61:41143)
I20260809 01:52:57.848409 14742 leader_election.cc:288] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149), 419b6368ced541299d700c2059e48866 (127.3.220.60:36917)
I20260809 01:52:57.848649 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.848130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56278)
0809 01:52:57.848353 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.848442 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
W20260809 01:52:57.848910 13953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:57.849169 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.848524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56284)
0809 01:52:57.848993 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.849066 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":81,"threads_started":1}
W20260809 01:52:57.849355 13677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:57.850278 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.850020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56286)
0809 01:52:57.850146 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.850206 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 01:52:57.850567 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.850250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56290)
0809 01:52:57.850429 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.850500 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260809 01:52:57.850713 13953 leader_election.cc:334] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149): Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
W20260809 01:52:57.850881 13677 leader_election.cc:334] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149): Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:57.853133 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.849032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.3.220.60:36917)
0809 01:52:57.849256 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:57.849261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:57.849274 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:52:57.849320 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:57.849323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:57.849362 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:57.849600 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:57.849863 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:57.850760 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:57.852080 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:57.852587 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:57.852711 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:57.852775 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:57.852938 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 01:52:57.853012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
I20260809 01:52:57.853137 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.848677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:36917 (local address 127.0.0.1:55750)
0809 01:52:57.848989 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.849003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:57.849092 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:57.849496 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:57.849499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:57.849571 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:57.849720 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:57.849727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:57.850913 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:57.850915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:57.851951 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:57.851960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:57.852703 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:57.852707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:57.852802 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:57.852824 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 01:52:57.852923 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 01:52:57.852997 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"spinlock_wait_cycles":4352,"thread_start_us":79,"threads_started":1}
I20260809 01:52:57.853278 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.848747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.3.220.61:41143)
0809 01:52:57.849475 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:57.849479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:57.849493 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:57.849562 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:57.849568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:57.849638 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:57.850017 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:57.850117 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:57.850958 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:57.852247 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:57.852726 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:57.852841 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:57.852896 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:57.853087 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 01:52:57.853167 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":625,"spinlock_wait_cycles":4992,"thread_start_us":62,"threads_started":1}
I20260809 01:52:57.853266 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.848291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:41143 (local address 127.0.0.1:37518)
0809 01:52:57.848642 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.848658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:57.848783 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:57.849777 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:57.849780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:57.849801 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:57.849901 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:57.849908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:57.851133 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:57.851136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:57.852126 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:57.852135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:57.852846 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:57.852849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:57.852941 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:57.852971 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 01:52:57.853069 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 01:52:57.853143 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
I20260809 01:52:57.853942 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "419b6368ced541299d700c2059e48866" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" is_pre_election: true
I20260809 01:52:57.853943 13975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "419b6368ced541299d700c2059e48866" is_pre_election: true
I20260809 01:52:57.854120 13975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8208ee9ce1d243faa7ee08a8c7177ada in term 1.
I20260809 01:52:57.854131 13713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419b6368ced541299d700c2059e48866 in term 1.
I20260809 01:52:57.854426 13679 leader_election.cc:302] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 419b6368ced541299d700c2059e48866, 8208ee9ce1d243faa7ee08a8c7177ada; no voters: 25c42ff3cd264b7082ea27fa4a0b42ef
I20260809 01:52:57.854460 13958 leader_election.cc:302] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 419b6368ced541299d700c2059e48866, 8208ee9ce1d243faa7ee08a8c7177ada; no voters: 25c42ff3cd264b7082ea27fa4a0b42ef
I20260809 01:52:57.854642 14742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 01:52:57.854727 14742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25c42ff3cd264b7082ea27fa4a0b42ef)
I20260809 01:52:57.854800 14742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:52:57.855829 14742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } }
I20260809 01:52:57.856184 14742 leader_election.cc:288] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 2 election: Requested vote from peers 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149), 419b6368ced541299d700c2059e48866 (127.3.220.60:36917)
I20260809 01:52:57.856321 14741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 01:52:57.856386 14741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25c42ff3cd264b7082ea27fa4a0b42ef)
I20260809 01:52:57.856436 14741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:52:57.856524 13975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "419b6368ced541299d700c2059e48866"
I20260809 01:52:57.856643 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.856330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56300)
0809 01:52:57.856531 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.856583 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260809 01:52:57.857283 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.857079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56314)
0809 01:52:57.857176 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.857224 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 01:52:57.857378 13677 leader_election.cc:334] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149): Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:57.857630 14741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } }
I20260809 01:52:57.857882 13975 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8208ee9ce1d243faa7ee08a8c7177ada in current term 2: Already voted for candidate 419b6368ced541299d700c2059e48866 in this term.
I20260809 01:52:57.858022 14741 leader_election.cc:288] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [CANDIDATE]: Term 2 election: Requested vote from peers 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149), 8208ee9ce1d243faa7ee08a8c7177ada (127.3.220.61:41143)
I20260809 01:52:57.858346 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.858150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56326)
0809 01:52:57.858255 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.858297 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 01:52:57.858520 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "419b6368ced541299d700c2059e48866" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8208ee9ce1d243faa7ee08a8c7177ada"
I20260809 01:52:57.858531 13679 leader_election.cc:302] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8208ee9ce1d243faa7ee08a8c7177ada; no voters: 25c42ff3cd264b7082ea27fa4a0b42ef, 419b6368ced541299d700c2059e48866
I20260809 01:52:57.858695 13713 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 419b6368ced541299d700c2059e48866 in current term 2: Already voted for candidate 8208ee9ce1d243faa7ee08a8c7177ada in this term.
I20260809 01:52:57.858768 14742 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 01:52:57.859340 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:57.859114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56332)
0809 01:52:57.859250 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:57.859292 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260809 01:52:57.859462 13953 leader_election.cc:334] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149): Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:57.859553 13953 leader_election.cc:302] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 419b6368ced541299d700c2059e48866; no voters: 25c42ff3cd264b7082ea27fa4a0b42ef, 8208ee9ce1d243faa7ee08a8c7177ada
I20260809 01:52:57.859691 14741 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 01:52:58.336184 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:58.335691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:36643)
0809 01:52:58.336002 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:58.336061 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":144,"threads_started":1}
I20260809 01:52:58.338515 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:58.338049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:50875)
0809 01:52:58.338320 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:58.338398 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":60,"threads_started":1}
I20260809 01:52:58.339148 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:58.338444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:58217)
0809 01:52:58.338986 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:58.339049 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":95,"threads_started":1}
I20260809 01:52:59.337723 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.337184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:45707)
0809 01:52:59.337497 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.337597 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260809 01:52:59.339524 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.339089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:51833)
0809 01:52:59.339317 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.339389 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260809 01:52:59.340288 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.339836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:42665)
0809 01:52:59.340137 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.340198 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20260809 01:52:59.404860 14763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:59.405038 14763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } }
I20260809 01:52:59.405530 14763 leader_election.cc:288] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149), 419b6368ced541299d700c2059e48866 (127.3.220.60:36917)
I20260809 01:52:59.406052 13975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "419b6368ced541299d700c2059e48866" is_pre_election: true
I20260809 01:52:59.406173 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.405804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56578)
0809 01:52:59.406025 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.406084 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260809 01:52:59.406224 13975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8208ee9ce1d243faa7ee08a8c7177ada in term 2.
I20260809 01:52:59.406605 13679 leader_election.cc:302] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419b6368ced541299d700c2059e48866, 8208ee9ce1d243faa7ee08a8c7177ada; no voters:
I20260809 01:52:59.406818 14763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 01:52:59.406903 14763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:59.406944 14763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 2 FOLLOWER]: Advancing to term 3
I20260809 01:52:59.407063 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.406803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56592)
0809 01:52:59.406929 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.406993 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 01:52:59.407224 13677 leader_election.cc:334] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149): Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:59.408154 14763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } }
I20260809 01:52:59.408501 14763 leader_election.cc:288] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 3 election: Requested vote from peers 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149), 419b6368ced541299d700c2059e48866 (127.3.220.60:36917)
I20260809 01:52:59.408849 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.408670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56594)
0809 01:52:59.408756 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.408799 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260809 01:52:59.408874 13975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "419b6368ced541299d700c2059e48866"
I20260809 01:52:59.409013 13975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 2 FOLLOWER]: Advancing to term 3
I20260809 01:52:59.409498 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.409299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56608)
0809 01:52:59.409401 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.409452 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 01:52:59.409622 13677 leader_election.cc:334] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149): Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
I20260809 01:52:59.410065 13975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8208ee9ce1d243faa7ee08a8c7177ada in term 3.
I20260809 01:52:59.410370 13679 leader_election.cc:302] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 419b6368ced541299d700c2059e48866, 8208ee9ce1d243faa7ee08a8c7177ada; no voters: 25c42ff3cd264b7082ea27fa4a0b42ef
I20260809 01:52:59.410567 14763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 3 FOLLOWER]: Leader election won for term 3
I20260809 01:52:59.410770 14763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [term 3 LEADER]: Becoming Leader. State: Replica: 8208ee9ce1d243faa7ee08a8c7177ada, State: Running, Role: LEADER
I20260809 01:52:59.411043 14763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } }
I20260809 01:52:59.412082 14766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8208ee9ce1d243faa7ee08a8c7177ada. Latest consensus state: current_term: 3 leader_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:59.412213 14766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:59.412511 14768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:52:59.414795 14768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:52:59.415182 14768 catalog_manager.cc:1259] Loaded cluster ID: b4288719cdb2436b861f8e28027d272e
I20260809 01:52:59.415227 14768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:52:59.415805 14768 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:52:59.416195 14768 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Loaded TSK: 0
I20260809 01:52:59.416888 14768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:52:59.447824 13700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:52:59.451062 13700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:52:59.454253 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.453969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56626)
0809 01:52:59.454088 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.454169 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 01:52:59.454370 13975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [term 3 FOLLOWER]: Refusing update from remote peer 8208ee9ce1d243faa7ee08a8c7177ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260809 01:52:59.454900 14763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 01:52:59.455051 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.454865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.0.0.1:56628)
0809 01:52:59.454949 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.455002 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260809 01:52:59.455256 13677 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada -> Peer 25c42ff3cd264b7082ea27fa4a0b42ef (127.3.220.62:45149): Couldn't send request to peer 25c42ff3cd264b7082ea27fa4a0b42ef. Status: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 01:52:59.456382 14771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8208ee9ce1d243faa7ee08a8c7177ada. Latest consensus state: current_term: 3 leader_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:59.456506 14771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:52:59.458184 14771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:59.458331 14771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419b6368ced541299d700c2059e48866 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:52:59.458576 14766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25c42ff3cd264b7082ea27fa4a0b42ef" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45149 } } peers { permanent_uuid: "8208ee9ce1d243faa7ee08a8c7177ada" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 41143 } attrs { promote: false } } peers { permanent_uuid: "419b6368ced541299d700c2059e48866" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 36917 } attrs { promote: false } } }
I20260809 01:52:59.458724 14766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada [sys.catalog]: This master's current role is: LEADER
I20260809 01:52:59.468883 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.464353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:37357 (local address 127.0.0.1:49628)
0809 01:52:59.464466 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.464482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:59.464574 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:59.465524 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:59.465558 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:59.465584 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:59.465735 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:59.465744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.466535 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.466542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:59.467609 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:59.467619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.468481 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.468484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:59.468571 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:59.468609 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 01:52:59.468679 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 01:52:59.468753 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":6}
I20260809 01:52:59.469353 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.464549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.3.220.1:37357)
0809 01:52:59.464840 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:59.464846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:59.464860 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:52:59.464916 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:59.464921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:59.465372 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:59.465506 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:59.465844 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.466451 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.467728 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.468350 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.468479 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:59.468548 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:59.469125 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 01:52:59.469219 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260809 01:52:59.470003 14136 tablet_service.cc:1467] Processing CreateTablet for tablet a45c61fec4bf41d88471a4a31eb787a9 (DEFAULT_TABLE table=test-table [id=0f9da6fc7d1945e6b1a2ea52ad19f31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:59.470057 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.464634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:36817 (local address 127.0.0.1:39090)
0809 01:52:59.464895 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.464908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:59.464991 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:59.465518 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:59.465521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:59.465571 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:59.465741 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:59.465750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.467513 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.467517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:59.468549 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:59.468558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.469630 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.469634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:59.469751 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:59.469783 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 01:52:59.469839 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 01:52:59.469917 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":42,"thread_start_us":108,"threads_started":1}
I20260809 01:52:59.470338 14136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45c61fec4bf41d88471a4a31eb787a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:59.470541 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.464998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.3.220.3:36817)
0809 01:52:59.465255 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:59.465271 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:59.465287 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:52:59.465342 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:59.465344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:59.465391 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:59.465654 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:59.466605 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.467408 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.468650 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.469334 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.470253 (+ 919us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:59.470316 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:59.470352 (+ 36us) server_negotiation.cc:299] Negotiation successful
0809 01:52:59.470415 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260809 01:52:59.471004 14523 tablet_service.cc:1467] Processing CreateTablet for tablet a45c61fec4bf41d88471a4a31eb787a9 (DEFAULT_TABLE table=test-table [id=0f9da6fc7d1945e6b1a2ea52ad19f31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:59.471230 14523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45c61fec4bf41d88471a4a31eb787a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:59.472662 14784 tablet_bootstrap.cc:492] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a: Bootstrap starting.
I20260809 01:52:59.473578 14785 tablet_bootstrap.cc:492] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: Bootstrap starting.
I20260809 01:52:59.473673 14784 tablet_bootstrap.cc:654] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:59.474498 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.465010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:43743 (local address 127.0.0.1:36080)
0809 01:52:59.466035 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.466046 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:59.466158 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:59.466399 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:52:59.466401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:52:59.466417 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:52:59.466496 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:59.466504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.468708 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.468711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:59.469818 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:52:59.469829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.474132 (+ 4303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.474136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:52:59.474225 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:59.474278 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 01:52:59.474338 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 01:52:59.474407 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":84,"threads_started":1}
I20260809 01:52:59.474568 14785 tablet_bootstrap.cc:654] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:59.474956 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.465132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.3.220.2:43743)
0809 01:52:59.465430 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:59.465435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:59.466125 (+ 690us) server_negotiation.cc:407] Connection header received
0809 01:52:59.466248 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:59.466251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:59.466296 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:59.466391 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:52:59.466623 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.467421 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.469950 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.470726 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.474702 (+ 3976us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:59.474755 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:59.474790 (+ 35us) server_negotiation.cc:299] Negotiation successful
0809 01:52:59.474849 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
I20260809 01:52:59.475544 14328 tablet_service.cc:1467] Processing CreateTablet for tablet a45c61fec4bf41d88471a4a31eb787a9 (DEFAULT_TABLE table=test-table [id=0f9da6fc7d1945e6b1a2ea52ad19f31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:52:59.475819 14328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45c61fec4bf41d88471a4a31eb787a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:52:59.475893 14785 tablet_bootstrap.cc:492] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: No bootstrap required, opened a new log
I20260809 01:52:59.475967 14785 ts_tablet_manager.cc:1397] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 01:52:59.475973 14784 tablet_bootstrap.cc:492] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a: No bootstrap required, opened a new log
I20260809 01:52:59.476034 14784 ts_tablet_manager.cc:1397] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 01:52:59.476398 14785 raft_consensus.cc:348] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.476365 14784 raft_consensus.cc:348] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.476504 14785 raft_consensus.cc:374] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:59.476517 14784 raft_consensus.cc:374] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:59.476532 14785 raft_consensus.cc:729] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd69de1d46fe4e24a3726b915c893e05, State: Initialized, Role: FOLLOWER
I20260809 01:52:59.476542 14784 raft_consensus.cc:729] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c782dcddc5764bd39e69380cc3ad640a, State: Initialized, Role: FOLLOWER
I20260809 01:52:59.476655 14785 consensus_queue.cc:260] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.476686 14784 consensus_queue.cc:260] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.477046 14785 ts_tablet_manager.cc:1428] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260809 01:52:59.477083 14784 ts_tablet_manager.cc:1428] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 01:52:59.479090 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.478549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:33685)
0809 01:52:59.478959 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.479019 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330}
I20260809 01:52:59.479962 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.479761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:51357)
0809 01:52:59.479860 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.479907 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 01:52:59.480528 14234 heartbeater.cc:503] Master 127.3.220.61:41143 was elected leader, sending a full tablet report...
I20260809 01:52:59.480736 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.480491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:56319)
0809 01:52:59.480628 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.480675 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260809 01:52:59.481563 14630 heartbeater.cc:503] Master 127.3.220.61:41143 was elected leader, sending a full tablet report...
I20260809 01:52:59.482262 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.482041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:58523)
0809 01:52:59.482142 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.482197 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260809 01:52:59.482663 14792 tablet_bootstrap.cc:492] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: Bootstrap starting.
I20260809 01:52:59.483505 14792 tablet_bootstrap.cc:654] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: Neither blocks nor log segments found. Creating new log.
I20260809 01:52:59.484766 14792 tablet_bootstrap.cc:492] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: No bootstrap required, opened a new log
W20260809 01:52:59.484812 14235 tablet.cc:2369] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:59.484867 14792 ts_tablet_manager.cc:1397] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 01:52:59.485301 14792 raft_consensus.cc:348] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.485407 14792 raft_consensus.cc:374] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:52:59.485430 14792 raft_consensus.cc:729] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f5e2bf2e0543a88e3d3d6783756314, State: Initialized, Role: FOLLOWER
I20260809 01:52:59.485612 14792 consensus_queue.cc:260] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.485994 14792 ts_tablet_manager.cc:1428] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260809 01:52:59.486564 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.486308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:45239)
0809 01:52:59.486434 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.486493 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 01:52:59.487391 14425 heartbeater.cc:503] Master 127.3.220.61:41143 was elected leader, sending a full tablet report...
I20260809 01:52:59.487696 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.487415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:43863)
0809 01:52:59.487577 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.487631 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 01:52:59.635787 14788 raft_consensus.cc:482] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:52:59.636013 14788 raft_consensus.cc:504] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.636519 14788 leader_election.cc:288] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357), 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743)
I20260809 01:52:59.637074 14164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45c61fec4bf41d88471a4a31eb787a9" candidate_uuid: "fd69de1d46fe4e24a3726b915c893e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c782dcddc5764bd39e69380cc3ad640a" is_pre_election: true
I20260809 01:52:59.637120 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45c61fec4bf41d88471a4a31eb787a9" candidate_uuid: "fd69de1d46fe4e24a3726b915c893e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f5e2bf2e0543a88e3d3d6783756314" is_pre_election: true
I20260809 01:52:59.637285 14359 raft_consensus.cc:2454] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd69de1d46fe4e24a3726b915c893e05 in term 0.
I20260809 01:52:59.637295 14164 raft_consensus.cc:2454] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd69de1d46fe4e24a3726b915c893e05 in term 0.
I20260809 01:52:59.637706 14461 leader_election.cc:302] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f5e2bf2e0543a88e3d3d6783756314, fd69de1d46fe4e24a3726b915c893e05; no voters:
I20260809 01:52:59.637928 14788 raft_consensus.cc:2781] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:52:59.637995 14788 raft_consensus.cc:482] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:52:59.638052 14788 raft_consensus.cc:3037] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:59.639096 14788 raft_consensus.cc:504] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.639456 14788 leader_election.cc:288] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [CANDIDATE]: Term 1 election: Requested vote from peers c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357), 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743)
I20260809 01:52:59.639816 14164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45c61fec4bf41d88471a4a31eb787a9" candidate_uuid: "fd69de1d46fe4e24a3726b915c893e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c782dcddc5764bd39e69380cc3ad640a"
I20260809 01:52:59.639856 14359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45c61fec4bf41d88471a4a31eb787a9" candidate_uuid: "fd69de1d46fe4e24a3726b915c893e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f5e2bf2e0543a88e3d3d6783756314"
I20260809 01:52:59.639950 14164 raft_consensus.cc:3037] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:59.639982 14359 raft_consensus.cc:3037] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:52:59.640879 14164 raft_consensus.cc:2454] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd69de1d46fe4e24a3726b915c893e05 in term 1.
I20260809 01:52:59.640919 14359 raft_consensus.cc:2454] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd69de1d46fe4e24a3726b915c893e05 in term 1.
I20260809 01:52:59.641196 14460 leader_election.cc:302] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c782dcddc5764bd39e69380cc3ad640a, fd69de1d46fe4e24a3726b915c893e05; no voters:
I20260809 01:52:59.641371 14788 raft_consensus.cc:2781] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:52:59.641655 14788 raft_consensus.cc:686] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 1 LEADER]: Becoming Leader. State: Replica: fd69de1d46fe4e24a3726b915c893e05, State: Running, Role: LEADER
I20260809 01:52:59.642059 14788 consensus_queue.cc:237] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } }
I20260809 01:52:59.642505 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.642226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:48687)
0809 01:52:59.642347 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.642414 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 01:52:59.644570 13699 catalog_manager.cc:5495] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd69de1d46fe4e24a3726b915c893e05 (127.3.220.3). New cstate: current_term: 1 leader_uuid: "fd69de1d46fe4e24a3726b915c893e05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd69de1d46fe4e24a3726b915c893e05" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 36817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 } health_report { overall_health: UNKNOWN } } }
I20260809 01:52:59.663545 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.660260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.3.220.3:36817)
0809 01:52:59.660405 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 01:52:59.660412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:52:59.660428 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:52:59.660598 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:52:59.660601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:52:59.660648 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:52:59.660747 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:52:59.661002 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.661571 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.662472 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:52:59.662703 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:52:59.662811 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:52:59.663100 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:52:59.663170 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:52:59.663251 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 01:52:59.663345 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 01:52:59.663399 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 01:52:59.665508 14631 tablet.cc:2369] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:52:59.665804 14164 raft_consensus.cc:1261] T a45c61fec4bf41d88471a4a31eb787a9 P c782dcddc5764bd39e69380cc3ad640a [term 1 FOLLOWER]: Refusing update from remote peer fd69de1d46fe4e24a3726b915c893e05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:59.665804 14359 raft_consensus.cc:1261] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 1 FOLLOWER]: Refusing update from remote peer fd69de1d46fe4e24a3726b915c893e05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:52:59.666317 14796 consensus_queue.cc:1035] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f5e2bf2e0543a88e3d3d6783756314" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:59.666483 14788 consensus_queue.cc:1035] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c782dcddc5764bd39e69380cc3ad640a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 37357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:52:59.668403 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.668066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:54491)
0809 01:52:59.668254 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.668324 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260809 01:52:59.668684 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.668380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:48863)
0809 01:52:59.668542 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.668597 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260809 01:52:59.670205 14800 mvcc.cc:205] Tried to move back new op lower bound from 7316440595106353152 to 7316440595015348224. Current Snapshot: MvccSnapshot[applied={T|T < 7316440595106353152}]
I20260809 01:52:59.671497 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.671225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:38555)
0809 01:52:59.671356 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.671413 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 01:52:59.674871 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.674424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.2:50739)
0809 01:52:59.674737 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.674797 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260809 01:52:59.675773 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.675384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.3:46059)
0809 01:52:59.675667 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.675715 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260809 01:52:59.675793 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:52:59.675468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45149 (local address 127.3.220.1:52859)
0809 01:52:59.675651 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.675703 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260809 01:52:59.676836 13699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:59.677000 13699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:52:59.681404 13699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8208ee9ce1d243faa7ee08a8c7177ada: Sending DeleteTablet for 3 replicas of tablet a45c61fec4bf41d88471a4a31eb787a9
I20260809 01:52:59.682099 14328 tablet_service.cc:1514] Processing DeleteTablet for tablet a45c61fec4bf41d88471a4a31eb787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:59 UTC) from {username='oryx', principal='oryx/127.3.220.61@KRBTEST.COM'} at 127.0.0.1:36080
I20260809 01:52:59.682120 14523 tablet_service.cc:1514] Processing DeleteTablet for tablet a45c61fec4bf41d88471a4a31eb787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:59 UTC) from {username='oryx', principal='oryx/127.3.220.61@KRBTEST.COM'} at 127.0.0.1:39090
I20260809 01:52:59.682410 14136 tablet_service.cc:1514] Processing DeleteTablet for tablet a45c61fec4bf41d88471a4a31eb787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:59 UTC) from {username='oryx', principal='oryx/127.3.220.61@KRBTEST.COM'} at 127.0.0.1:49628
I20260809 01:52:59.682428 14132 tablet_service.cc:1514] Processing DeleteTablet for tablet a45c61fec4bf41d88471a4a31eb787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:52:59 UTC) from {username='oryx', principal='oryx/127.3.220.61@KRBTEST.COM'} at 127.0.0.1:49628
I20260809 01:52:59.682884 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 13908
I20260809 01:52:59.693687 14816 tablet_replica.cc:330] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: stopping tablet replica
I20260809 01:52:59.693877 14816 raft_consensus.cc:2229] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:52:59.694063 14816 raft_consensus.cc:2258] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:59.694398 14815 tablet_replica.cc:330] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: stopping tablet replica
I20260809 01:52:59.694566 14815 raft_consensus.cc:2229] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:52:59.694806 14815 raft_consensus.cc:2258] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:52:59.695037 14816 ts_tablet_manager.cc:1905] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:59.695509 14815 ts_tablet_manager.cc:1905] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:52:59.697827 14815 ts_tablet_manager.cc:1918] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:59.697925 14815 log.cc:1196] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/a45c61fec4bf41d88471a4a31eb787a9
I20260809 01:52:59.698261 14815 ts_tablet_manager.cc:1939] T a45c61fec4bf41d88471a4a31eb787a9 P fd69de1d46fe4e24a3726b915c893e05: Deleting consensus metadata
I20260809 01:52:59.698407 14816 ts_tablet_manager.cc:1918] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:52:59.698500 14816 log.cc:1196] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/a45c61fec4bf41d88471a4a31eb787a9
I20260809 01:52:59.698819 14816 ts_tablet_manager.cc:1939] T a45c61fec4bf41d88471a4a31eb787a9 P 39f5e2bf2e0543a88e3d3d6783756314: Deleting consensus metadata
I20260809 01:52:59.699380 13679 catalog_manager.cc:4841] TS fd69de1d46fe4e24a3726b915c893e05 (127.3.220.3:36817): tablet a45c61fec4bf41d88471a4a31eb787a9 (table test-table [id=0f9da6fc7d1945e6b1a2ea52ad19f31f]) successfully deleted
I20260809 01:52:59.699534 13681 catalog_manager.cc:4841] TS 39f5e2bf2e0543a88e3d3d6783756314 (127.3.220.2:43743): tablet a45c61fec4bf41d88471a4a31eb787a9 (table test-table [id=0f9da6fc7d1945e6b1a2ea52ad19f31f]) successfully deleted
W20260809 01:52:59.699683 14781 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:52:59.699194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:37357 (local address 127.0.0.1:49642)
0809 01:52:59.699329 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 01:52:59.699346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:52:59.699446 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:52:59.699555 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:37357: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 01:52:59.699891 13679 catalog_manager.cc:4584] TS c782dcddc5764bd39e69380cc3ad640a (127.3.220.1:37357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a45c61fec4bf41d88471a4a31eb787a9: Network error: Client connection negotiation failed: client connection to 127.3.220.1:37357: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260809 01:52:59.699945 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 14240
I20260809 01:52:59.712905 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 14433
I20260809 01:52:59.727635 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 13482
I20260809 01:52:59.736670 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 13638
2026-08-09T01:52:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7287 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:52:59 dist-test-slave-gkw7 krb5kdc[14821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:52:59 dist-test-slave-gkw7 krb5kdc[14821](info): set up 2 sockets
Aug 09 01:52:59 dist-test-slave-gkw7 krb5kdc[14821](info): commencing operation
krb5kdc: starting...
W20260809 01:53:00.968801 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:53:01 dist-test-slave-gkw7 krb5kdc[14821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:53:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.62.keytab.
Entry for principal oryx/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.62.keytab.
I20260809 01:53:01.105443 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:42101
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.62.keytab
--principal=oryx/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:01.223591 14836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:01.223949 14836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:01.224043 14836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:01.228199 14836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:53:01.228305 14836 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 01:53:01.228343 14836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:01.228415 14836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:01.228461 14836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:53:01.228502 14836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:53:01.234063 14836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:42101
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.62.keytab
--principal=oryx/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:01.235234 14836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:01.236353 14836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:01.245391 14843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:01.245433 14845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:01.245433 14841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:01.245747 14836 server_base.cc:1034] running on GCE node
I20260809 01:53:01.246515 14836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:01.247782 14836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:01.248999 14836 hybrid_clock.cc:648] HybridClock initialized: now 1786240381248976 us; error 39 us; skew 500 ppm
Aug 09 01:53:01 dist-test-slave-gkw7 krb5kdc[14821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:01.252321 14836 init.cc:377] Logged in from keytab as oryx/127.3.220.62@KRBTEST.COM (short username oryx)
I20260809 01:53:01.253474 14836 webserver.cc:460] Webserver started at http://127.3.220.62:39839/ using document root <none> and password file <none>
I20260809 01:53:01.254166 14836 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:01.254287 14836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:01.254561 14836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:01.256896 14836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "569dd80c966042ea9a9ebf5caeee32a0"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "0e425d01572ac59c99a80907fb250e6b"
server_key_iv: "15c93d035d6ba6858252d163e06f62d0"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.257629 14836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "569dd80c966042ea9a9ebf5caeee32a0"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "0e425d01572ac59c99a80907fb250e6b"
server_key_iv: "15c93d035d6ba6858252d163e06f62d0"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.262287 14836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 01:53:01.265013 14858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:01.266469 14836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:53:01.266630 14836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "569dd80c966042ea9a9ebf5caeee32a0"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "0e425d01572ac59c99a80907fb250e6b"
server_key_iv: "15c93d035d6ba6858252d163e06f62d0"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.266768 14836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:01.287011 14836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:01.290529 14836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:01.290788 14836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:01.300083 14836 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:42101
I20260809 01:53:01.300104 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:42101 every 8 connection(s)
I20260809 01:53:01.301235 14836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:53:01.303493 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 14836
I20260809 01:53:01.303686 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:53:01.303994 3952 external_mini_cluster.cc:1428] Setting key 2468772b7d00efb6b382232dd10f2441
I20260809 01:53:01.304556 14957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:01.312408 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: Bootstrap starting.
Aug 09 01:53:01 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:53:01.315353 14957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:01.316380 14957 log.cc:824] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:01.319123 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: No bootstrap required, opened a new log
I20260809 01:53:01.321382 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.305602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.3.220.62:42101)
0809 01:53:01.306096 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:01.306122 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:01.306172 (+ 50us) server_negotiation.cc:407] Connection header received
0809 01:53:01.307038 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:01.307063 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:01.307522 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:01.307929 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:01.308700 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.310287 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.311210 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.311922 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.312119 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.315018 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:01.315025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:01.315042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:01.315078 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:01.318029 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.318500 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.318506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.318512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.318716 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.318995 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.318998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.319000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.319553 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:01.319756 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:01.320028 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 01:53:01.320351 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":119,"threads_started":1}
I20260809 01:53:01.323314 14957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569dd80c966042ea9a9ebf5caeee32a0" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42101 } }
I20260809 01:53:01.323581 14957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:01.323714 14957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569dd80c966042ea9a9ebf5caeee32a0, State: Initialized, Role: FOLLOWER
I20260809 01:53:01.324151 14957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569dd80c966042ea9a9ebf5caeee32a0" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42101 } }
I20260809 01:53:01.324308 14957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:53:01.324383 14957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:53:01.324517 14957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:01.325732 14957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569dd80c966042ea9a9ebf5caeee32a0" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42101 } }
I20260809 01:53:01.326124 14957 leader_election.cc:302] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 569dd80c966042ea9a9ebf5caeee32a0; no voters:
I20260809 01:53:01.326489 14957 leader_election.cc:288] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:53:01.326658 14965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:01.326913 14965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [term 1 LEADER]: Becoming Leader. State: Replica: 569dd80c966042ea9a9ebf5caeee32a0, State: Running, Role: LEADER
I20260809 01:53:01.327328 14965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569dd80c966042ea9a9ebf5caeee32a0" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42101 } }
I20260809 01:53:01.327841 14957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:53:01.329072 14966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "569dd80c966042ea9a9ebf5caeee32a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569dd80c966042ea9a9ebf5caeee32a0" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42101 } } }
I20260809 01:53:01.329218 14966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:01.329396 14967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569dd80c966042ea9a9ebf5caeee32a0. Latest consensus state: current_term: 1 leader_uuid: "569dd80c966042ea9a9ebf5caeee32a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569dd80c966042ea9a9ebf5caeee32a0" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42101 } } }
I20260809 01:53:01.329492 14967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:01.329844 14975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:53:01.332664 14975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:53:01.339098 14975 catalog_manager.cc:1347] Generated new cluster ID: 35f04ea16adc4996a2d5211eac70b991
I20260809 01:53:01.339203 14975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:53:01.355293 14975 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:53:01.356581 14975 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:53:01.371083 14975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: Generated new TSK 0
I20260809 01:53:01.372045 14975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab.
Entry for principal oryx/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab.
I20260809 01:53:01.404480 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:42101
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab
--principal=oryx/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:01.519510 14995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:01.519785 14995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:01.519829 14995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:01.523952 14995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:01.524019 14995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:01.524056 14995 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:53:01.524127 14995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:53:01.529765 14995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab
--principal=oryx/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:01.530851 14995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:01.531924 14995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:01.541786 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:01.541795 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:01.541891 14995 server_base.cc:1034] running on GCE node
W20260809 01:53:01.541795 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:01.543279 14995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:01.544003 14995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:01.545234 14995 hybrid_clock.cc:648] HybridClock initialized: now 1786240381545214 us; error 31 us; skew 500 ppm
Aug 09 01:53:01 dist-test-slave-gkw7 krb5kdc[14821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:01.548171 14995 init.cc:377] Logged in from keytab as oryx/127.3.220.1@KRBTEST.COM (short username oryx)
I20260809 01:53:01.549077 14995 webserver.cc:460] Webserver started at http://127.3.220.1:37711/ using document root <none> and password file <none>
I20260809 01:53:01.549722 14995 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:01.549816 14995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:01.550051 14995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:01.552109 14995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "6ea5f5770434470d90b14f06c0884e53"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "539e52000fe6bb64d4294a5cbf05e620"
server_key_iv: "9c5000c8b31c2e3e0c990b19b3abe5c1"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.552692 14995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "6ea5f5770434470d90b14f06c0884e53"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "539e52000fe6bb64d4294a5cbf05e620"
server_key_iv: "9c5000c8b31c2e3e0c990b19b3abe5c1"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.556515 14995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260809 01:53:01.559090 15014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:01.560376 14995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 01:53:01.560513 14995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "6ea5f5770434470d90b14f06c0884e53"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "539e52000fe6bb64d4294a5cbf05e620"
server_key_iv: "9c5000c8b31c2e3e0c990b19b3abe5c1"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.560624 14995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:01.581661 14995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:01.584847 14995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:01.585071 14995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:01.586639 14995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:01.586723 14995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:01.586797 14995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:01.586848 14995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 01:53:01 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:53:01.598152 14995 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:44109
I20260809 01:53:01.598229 15184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:44109 every 8 connection(s)
I20260809 01:53:01.599303 14995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:53:01.600629 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.587517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:55311 (local address 127.3.220.62:42101)
0809 01:53:01.587692 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:01.587698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:01.588519 (+ 821us) server_negotiation.cc:407] Connection header received
0809 01:53:01.589800 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:01.589806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:01.589866 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:01.589924 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:01.591270 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.591904 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.593455 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.593753 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.593872 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.596690 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:01.596695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:01.596700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:01.596738 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:01.598479 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.598973 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.598977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.598979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.599069 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.599440 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.599444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.599445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.599695 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:01.599810 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:01.600383 (+ 573us) server_negotiation.cc:299] Negotiation successful
0809 01:53:01.600506 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 01:53:01.601480 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.587918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42101 (local address 127.3.220.1:55311)
0809 01:53:01.588353 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:01.588404 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:01.589495 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:01.590251 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:01.590257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:01.590583 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:01.591123 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:01.591146 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.592044 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.592048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:01.593322 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:01.593332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.593993 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.593997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:01.594149 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.594808 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:01.594824 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:01.596526 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:01.598591 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:01.598596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:01.598607 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:01.598860 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:01.599165 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:01.599167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:01.599168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:01.599312 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:01.599813 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:01.599819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:01.600097 (+ 278us) client_negotiation.cc:769] Sending connection context
0809 01:53:01.600328 (+ 231us) client_negotiation.cc:240] Negotiation successful
0809 01:53:01.600620 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":126,"threads_started":1}
I20260809 01:53:01.602267 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 14995
I20260809 01:53:01.602417 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:53:01.602762 3952 external_mini_cluster.cc:1428] Setting key 79b4782a25cc914efe036076952fcc0a
I20260809 01:53:01.603040 15185 heartbeater.cc:347] Connected to a master server at 127.3.220.62:42101
I20260809 01:53:01.603176 15185 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:01.603760 15185 heartbeater.cc:511] Master 127.3.220.62:42101 requested a full tablet report, sending...
I20260809 01:53:01.605461 14883 ts_manager.cc:194] Registered new tserver with Master: 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109)
I20260809 01:53:01.607043 14883 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:55311
I20260809 01:53:01.612924 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.605477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:43991 (local address 127.3.220.62:42101)
0809 01:53:01.605620 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:01.605626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:01.605642 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:53:01.605676 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:01.605679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:01.605759 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:01.605834 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:01.606366 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.606904 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.607839 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.608235 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.608383 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.609609 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:01.609613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:01.609616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:01.609641 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:01.611087 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.611560 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.611565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.611567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.611675 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.611964 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.611968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.611970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.612229 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:01.612360 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:01.612630 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 01:53:01.612750 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":49}
I20260809 01:53:01.612933 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.605182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42101 (local address 127.3.220.1:43991)
0809 01:53:01.605462 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:01.605476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:01.605584 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:01.605960 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:01.605965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:01.606141 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:01.606249 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:01.606254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.607038 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.607041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:01.607645 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:01.607656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.608383 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.608387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:01.608489 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.609093 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:01.609105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:01.609439 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:01.611200 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:01.611205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:01.611207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:01.611420 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:01.611781 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:01.611783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:01.611784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:01.611866 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:01.612362 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:01.612365 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:01.612461 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 01:53:01.612571 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 01:53:01.612738 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":37,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for oryx/127.3.220.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab.
Entry for principal oryx/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab.
I20260809 01:53:01.636904 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:42101
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab
--principal=oryx/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:01.751895 15196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:01.752146 15196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:01.752189 15196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:01.756021 15196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:01.756069 15196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:01.756105 15196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:53:01.756136 15196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:53:01.761088 15196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab
--principal=oryx/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:01.762090 15196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:01.763047 15196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:01.772925 15205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:01.772965 15202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:01.772969 15201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:01.773221 15196 server_base.cc:1034] running on GCE node
I20260809 01:53:01.773748 15196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:01.774454 15196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:01.775645 15196 hybrid_clock.cc:648] HybridClock initialized: now 1786240381775636 us; error 36 us; skew 500 ppm
Aug 09 01:53:01 dist-test-slave-gkw7 krb5kdc[14821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:01.778483 15196 init.cc:377] Logged in from keytab as oryx/127.3.220.2@KRBTEST.COM (short username oryx)
I20260809 01:53:01.779357 15196 webserver.cc:460] Webserver started at http://127.3.220.2:33963/ using document root <none> and password file <none>
I20260809 01:53:01.779884 15196 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:01.779943 15196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:01.780110 15196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:01.782231 15196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "a024199de6b147fbac4a3f8292166fe7"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "0222cc3f79cb0ef193a205d0072d0ef9"
server_key_iv: "bb4e657f79af8d254a2c50e0460e69dc"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.782750 15196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "a024199de6b147fbac4a3f8292166fe7"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "0222cc3f79cb0ef193a205d0072d0ef9"
server_key_iv: "bb4e657f79af8d254a2c50e0460e69dc"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.786443 15196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 01:53:01.788700 15218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:01.789978 15196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:53:01.790081 15196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "a024199de6b147fbac4a3f8292166fe7"
format_stamp: "Formatted at 2026-08-09 01:53:01 on dist-test-slave-gkw7"
server_key: "0222cc3f79cb0ef193a205d0072d0ef9"
server_key_iv: "bb4e657f79af8d254a2c50e0460e69dc"
server_key_version: "encryptionkey@0"
I20260809 01:53:01.790163 15196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:01.806680 15196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:01.809474 15196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:01.809737 15196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:01.811269 15196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:01.811326 15196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:01.811367 15196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:01.811424 15196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 01:53:01 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:53:01.823457 15196 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:46789
I20260809 01:53:01.823527 15383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:46789 every 8 connection(s)
I20260809 01:53:01.824494 15196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:53:01.826184 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.812013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:47075 (local address 127.3.220.62:42101)
0809 01:53:01.812171 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:01.812176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:01.813071 (+ 895us) server_negotiation.cc:407] Connection header received
0809 01:53:01.814153 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:01.814159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:01.814223 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:01.814293 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:01.815905 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.816774 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.818345 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.818583 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.818714 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.822186 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:01.822191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:01.822194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:01.822231 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:01.823890 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.824444 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.824447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.824449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.824534 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.824897 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.824901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.824902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.825160 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:01.825259 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:01.825902 (+ 643us) server_negotiation.cc:299] Negotiation successful
0809 01:53:01.826021 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:53:01.826854 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.812442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42101 (local address 127.3.220.2:47075)
0809 01:53:01.812894 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:01.812943 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:01.813918 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:01.814510 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:01.814521 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:01.815050 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:01.815691 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:01.815726 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.816901 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.816905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:01.818210 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:01.818220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.818814 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.818823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:01.819070 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.819821 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:01.819850 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:01.822002 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:01.824012 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:01.824019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:01.824034 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:01.824329 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:01.824630 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:01.824634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:01.824636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:01.824789 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:01.825316 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:01.825323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:01.825633 (+ 310us) client_negotiation.cc:769] Sending connection context
0809 01:53:01.825794 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 01:53:01.826042 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":137,"threads_started":1}
I20260809 01:53:01.828277 15385 heartbeater.cc:347] Connected to a master server at 127.3.220.62:42101
I20260809 01:53:01.828441 15385 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:01.829038 15385 heartbeater.cc:511] Master 127.3.220.62:42101 requested a full tablet report, sending...
I20260809 01:53:01.830471 14883 ts_manager.cc:194] Registered new tserver with Master: a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789)
I20260809 01:53:01.831311 14883 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:47075
I20260809 01:53:01.834668 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 15196
I20260809 01:53:01.834765 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:53:01.835108 3952 external_mini_cluster.cc:1428] Setting key 2808e61553e124dbb9882ffa2d0724d3
I20260809 01:53:01.838867 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.830655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42101 (local address 127.3.220.2:59791)
0809 01:53:01.830981 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:01.830993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:01.831081 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:01.831308 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:01.831310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:01.831537 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:01.831685 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:01.831694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.832539 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.832542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:01.833367 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:01.833378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.833906 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.833908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:01.833980 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.834566 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:01.834587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:01.835004 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:01.837007 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:01.837013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:01.837018 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:01.837298 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:01.837627 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:01.837633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:01.837637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:01.837746 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:01.838157 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:01.838162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:01.838292 (+ 130us) client_negotiation.cc:769] Sending connection context
0809 01:53:01.838411 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 01:53:01.838589 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260809 01:53:01.839707 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:01.830718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:59791 (local address 127.3.220.62:42101)
0809 01:53:01.830840 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:01.830844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:01.831077 (+ 233us) server_negotiation.cc:407] Connection header received
0809 01:53:01.831142 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:01.831145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:01.831192 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:01.831286 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:01.831808 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.832393 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.833492 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:01.833795 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:01.833889 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:01.835145 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:01.835149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:01.835153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:01.835181 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:01.836885 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.837405 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.837407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.837409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.837484 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:01.837842 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:01.837844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:01.837845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:01.838052 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:01.838145 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:01.838474 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 01:53:01.838607 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for oryx/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab.
Entry for principal oryx/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab.
I20260809 01:53:01.865430 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:42101
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab
--principal=oryx/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:01.981812 15397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:01.982164 15397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:01.982242 15397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:01.986418 15397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:01.986517 15397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:01.986588 15397 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 01:53:01.986672 15397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:53:01.992213 15397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab
--principal=oryx/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.220.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:01.993506 15397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:01.994746 15397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:53:02.005157 15397 server_base.cc:1034] running on GCE node
W20260809 01:53:02.005103 15410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:02.005136 15409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:02.005095 15412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:02.005897 15397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:02.006613 15397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:02.007812 15397 hybrid_clock.cc:648] HybridClock initialized: now 1786240382007795 us; error 38 us; skew 500 ppm
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:02.010953 15397 init.cc:377] Logged in from keytab as oryx/127.3.220.3@KRBTEST.COM (short username oryx)
I20260809 01:53:02.011992 15397 webserver.cc:460] Webserver started at http://127.3.220.3:44497/ using document root <none> and password file <none>
I20260809 01:53:02.012602 15397 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:02.012699 15397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:02.012951 15397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:02.015162 15397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "c0673839c2c44e13b2b9955b60f3f9d8"
format_stamp: "Formatted at 2026-08-09 01:53:02 on dist-test-slave-gkw7"
server_key: "9a810baf8339e91dabff5c6bb5dd3f68"
server_key_iv: "98951c3adda490521330bb4e308823f6"
server_key_version: "encryptionkey@0"
I20260809 01:53:02.015754 15397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "c0673839c2c44e13b2b9955b60f3f9d8"
format_stamp: "Formatted at 2026-08-09 01:53:02 on dist-test-slave-gkw7"
server_key: "9a810baf8339e91dabff5c6bb5dd3f68"
server_key_iv: "98951c3adda490521330bb4e308823f6"
server_key_version: "encryptionkey@0"
I20260809 01:53:02.019747 15397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260809 01:53:02.022437 15423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:02.023694 15397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:53:02.023840 15397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "c0673839c2c44e13b2b9955b60f3f9d8"
format_stamp: "Formatted at 2026-08-09 01:53:02 on dist-test-slave-gkw7"
server_key: "9a810baf8339e91dabff5c6bb5dd3f68"
server_key_iv: "98951c3adda490521330bb4e308823f6"
server_key_version: "encryptionkey@0"
I20260809 01:53:02.023967 15397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:02.042397 15397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:02.045902 15397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:02.046125 15397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:02.047638 15397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:02.047715 15397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:02.047792 15397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:02.047839 15397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240382, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:53:02.059481 15397 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:39969
I20260809 01:53:02.059556 15572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:39969 every 8 connection(s)
I20260809 01:53:02.060639 15397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:53:02.062629 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.048635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:48005 (local address 127.3.220.62:42101)
0809 01:53:02.048779 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.048783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.049965 (+ 1182us) server_negotiation.cc:407] Connection header received
0809 01:53:02.051118 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.051122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.051185 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.051284 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:02.053026 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.053912 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.055344 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.055586 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.055715 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.058470 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:02.058474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:02.058477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:02.058506 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:02.060301 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.060861 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.060865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.060867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.060968 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.061373 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.061376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.061378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.061631 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:02.061765 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.062330 (+ 565us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.062460 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 01:53:02.062858 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 15397
I20260809 01:53:02.062989 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:53:02.063366 3952 external_mini_cluster.cc:1428] Setting key b0ab2185a913c33781d576419ff71542
I20260809 01:53:02.063619 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.049217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42101 (local address 127.3.220.3:48005)
0809 01:53:02.049783 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.049829 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.050870 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.051523 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.051533 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.052045 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:02.052837 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.052862 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.054045 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.054048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.055216 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.055227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.055778 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.055782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.055975 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.056655 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:02.056671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:02.058286 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:02.060426 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:02.060430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:02.060442 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:02.060748 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:02.061079 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:02.061082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:02.061084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:02.061253 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:02.061761 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:02.061768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:02.062020 (+ 252us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.062286 (+ 266us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.062631 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":130,"threads_started":1}
I20260809 01:53:02.064884 15573 heartbeater.cc:347] Connected to a master server at 127.3.220.62:42101
I20260809 01:53:02.065057 15573 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:02.065698 15573 heartbeater.cc:511] Master 127.3.220.62:42101 requested a full tablet report, sending...
I20260809 01:53:02.066780 14879 ts_manager.cc:194] Registered new tserver with Master: c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969)
I20260809 01:53:02.067569 14879 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:48005
I20260809 01:53:02.074375 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.067343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:36933 (local address 127.3.220.62:42101)
0809 01:53:02.067871 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.067874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.067884 (+ 10us) server_negotiation.cc:407] Connection header received
0809 01:53:02.067911 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.067912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.067942 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.068032 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:02.068488 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.068999 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.069920 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.070158 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.070282 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.071409 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:02.071413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:02.071416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:02.071440 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:02.072658 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.073108 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.073112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.073114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.073199 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.073483 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.073487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.073488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.073712 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:02.073823 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.074096 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.074214 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470}
I20260809 01:53:02.074388 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.067203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42101 (local address 127.3.220.3:36933)
0809 01:53:02.067531 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.067547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.067641 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.068051 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.068053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.068266 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:02.068370 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.068376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.069129 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.069132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.069801 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.069810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.070283 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.070287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.070380 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.070848 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:02.070859 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:02.071273 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:02.072791 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:02.072793 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:02.072796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:02.073001 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:02.073294 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:02.073297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:02.073299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:02.073387 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:02.073813 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:02.073816 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:02.073918 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.074026 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.074177 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":115,"threads_started":1}
I20260809 01:53:02.077662 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 01:53:02.103734 14963 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:02.096808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.3.220.62:42101)
0809 01:53:02.096946 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.096950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.096962 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:53:02.097004 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.097006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.097046 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.097112 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:02.098060 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.098593 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.099435 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.099648 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.099735 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.103566 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54406: BlockingRecv error: recv got EOF from 127.0.0.1:54406 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:53:02.112802 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.105153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.3.220.62:42101)
0809 01:53:02.105281 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.105284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.105435 (+ 151us) server_negotiation.cc:407] Connection header received
0809 01:53:02.105572 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.105577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.105623 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.105698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:02.106270 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.106767 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.107499 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.107703 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.107781 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.110068 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:02.110071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:02.110073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:02.110096 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:02.111393 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.111766 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.111769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.111770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.111862 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.112091 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.112093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.112094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.112287 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:02.112390 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.112581 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.112679 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:53:02.116083 14879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:53:02.119174 14879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:53:02.134598 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.127719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:44109 (local address 127.0.0.1:38218)
0809 01:53:02.128316 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.128375 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.128504 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.129797 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.129803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.129826 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:02.130016 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.130023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.131280 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.131284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.132860 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.132868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.133986 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.133990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.134116 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.134156 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.134262 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.134404 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":98,"thread_start_us":107,"threads_started":1}
I20260809 01:53:02.134745 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.129235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.3.220.2:46789)
0809 01:53:02.129573 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.129585 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.129895 (+ 310us) server_negotiation.cc:407] Connection header received
0809 01:53:02.129979 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.129985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.130182 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.130346 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:02.130660 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.131733 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.133149 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.133779 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.133908 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.134114 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.134497 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.134593 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":120,"threads_started":1}
I20260809 01:53:02.134600 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.127965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:39969 (local address 127.0.0.1:56718)
0809 01:53:02.128563 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.128576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.128662 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.130207 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.130210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.130225 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:02.130353 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.130359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.131801 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.131806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.132695 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.132705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.134108 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.134112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.134216 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.134239 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.134308 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.134413 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":102,"thread_start_us":104,"threads_started":1}
I20260809 01:53:02.135281 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.129285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.3.220.3:39969)
0809 01:53:02.129774 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.129786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.129808 (+ 22us) server_negotiation.cc:407] Connection header received
0809 01:53:02.129882 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.129886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.130063 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.130279 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:02.130483 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.131651 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.133176 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.133985 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.134092 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.134990 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.135042 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.135135 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":108,"threads_started":1}
I20260809 01:53:02.134817 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.128479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:46789 (local address 127.0.0.1:50338)
0809 01:53:02.129679 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.129690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.129800 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.130431 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.130434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.130452 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:02.130538 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.130543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.131847 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.131851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.133030 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.133041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.134167 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.134170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.134240 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.134256 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.134331 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.134413 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":28,"thread_start_us":98,"threads_started":1}
I20260809 01:53:02.135651 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.128857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.3.220.1:44109)
0809 01:53:02.129191 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.129199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.129227 (+ 28us) server_negotiation.cc:407] Connection header received
0809 01:53:02.129344 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.129352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.129620 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.129790 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:02.130141 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.131139 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.132970 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.133840 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.135204 (+ 1364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.135366 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.135422 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.135522 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
I20260809 01:53:02.138500 15492 tablet_service.cc:1467] Processing CreateTablet for tablet 379f0cfaf959419b9a0f6855e600a183 (DEFAULT_TABLE table=test-table [id=d98886cb05dd4869bb20edd4b5e8b24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.139035 15091 tablet_service.cc:1467] Processing CreateTablet for tablet 379f0cfaf959419b9a0f6855e600a183 (DEFAULT_TABLE table=test-table [id=d98886cb05dd4869bb20edd4b5e8b24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.139678 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379f0cfaf959419b9a0f6855e600a183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.139472 15286 tablet_service.cc:1467] Processing CreateTablet for tablet 379f0cfaf959419b9a0f6855e600a183 (DEFAULT_TABLE table=test-table [id=d98886cb05dd4869bb20edd4b5e8b24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.140034 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379f0cfaf959419b9a0f6855e600a183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.140403 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379f0cfaf959419b9a0f6855e600a183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.145848 15610 tablet_bootstrap.cc:492] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: Bootstrap starting.
I20260809 01:53:02.147929 15612 tablet_bootstrap.cc:492] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Bootstrap starting.
I20260809 01:53:02.148295 15610 tablet_bootstrap.cc:654] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.148360 15611 tablet_bootstrap.cc:492] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Bootstrap starting.
I20260809 01:53:02.149194 15610 log.cc:824] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:02.150547 15611 tablet_bootstrap.cc:654] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.151103 15610 tablet_bootstrap.cc:492] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: No bootstrap required, opened a new log
I20260809 01:53:02.151328 15610 ts_tablet_manager.cc:1397] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 01:53:02.151638 15612 tablet_bootstrap.cc:654] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.151387 15611 log.cc:824] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:02.152537 15612 log.cc:824] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:02.154193 15611 tablet_bootstrap.cc:492] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: No bootstrap required, opened a new log
I20260809 01:53:02.154223 15612 tablet_bootstrap.cc:492] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: No bootstrap required, opened a new log
I20260809 01:53:02.154369 15612 ts_tablet_manager.cc:1397] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 01:53:02.154389 15611 ts_tablet_manager.cc:1397] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:53:02.156132 15610 raft_consensus.cc:348] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.156416 15610 raft_consensus.cc:374] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.156467 15610 raft_consensus.cc:729] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5f5770434470d90b14f06c0884e53, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.156924 15610 consensus_queue.cc:260] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.157157 15612 raft_consensus.cc:348] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.157639 15185 heartbeater.cc:503] Master 127.3.220.62:42101 was elected leader, sending a full tablet report...
I20260809 01:53:02.157968 15610 ts_tablet_manager.cc:1428] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260809 01:53:02.158315 15612 raft_consensus.cc:374] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.158440 15612 raft_consensus.cc:729] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a024199de6b147fbac4a3f8292166fe7, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.158890 15612 consensus_queue.cc:260] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.158812 15611 raft_consensus.cc:348] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.159044 15611 raft_consensus.cc:374] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.159154 15611 raft_consensus.cc:729] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0673839c2c44e13b2b9955b60f3f9d8, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.159582 15385 heartbeater.cc:503] Master 127.3.220.62:42101 was elected leader, sending a full tablet report...
I20260809 01:53:02.159544 15611 consensus_queue.cc:260] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.159891 15612 ts_tablet_manager.cc:1428] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:53:02.160331 15573 heartbeater.cc:503] Master 127.3.220.62:42101 was elected leader, sending a full tablet report...
I20260809 01:53:02.160487 15611 ts_tablet_manager.cc:1428] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:53:02.293447 15619 raft_consensus.cc:482] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:53:02.293776 15619 raft_consensus.cc:504] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.295087 15619 leader_election.cc:288] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109), a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789)
I20260809 01:53:02.299067 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.295379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:46789 (local address 127.3.220.3:39701)
0809 01:53:02.295804 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.295815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.295925 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.296163 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.296166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.296181 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:02.296316 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.296321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.297041 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.297045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.297969 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.297978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.298615 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.298618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.298722 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.298751 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.298842 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.298911 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":167,"threads_started":1}
I20260809 01:53:02.299052 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.295491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:39701 (local address 127.3.220.2:46789)
0809 01:53:02.295624 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.295628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.295894 (+ 266us) server_negotiation.cc:407] Connection header received
0809 01:53:02.296012 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.296015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.296067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.296152 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:02.296432 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.296926 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.298092 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.298491 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.298610 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.298678 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.298859 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.298931 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:53:02.299263 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.295397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:51641 (local address 127.3.220.1:44109)
0809 01:53:02.295535 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.295541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.295622 (+ 81us) server_negotiation.cc:407] Connection header received
0809 01:53:02.295817 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.295819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.295870 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.295973 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:02.296248 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.296765 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.298267 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.298711 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.298836 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.298899 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.299083 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.299152 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 01:53:02.299279 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.295325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:44109 (local address 127.3.220.3:51641)
0809 01:53:02.295534 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.295550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.295667 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.295988 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.295991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.296011 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:02.296132 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.296138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.296884 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.296887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.298129 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.298139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.298837 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.298841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.298953 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.298970 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.299053 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.299131 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260809 01:53:02.299994 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379f0cfaf959419b9a0f6855e600a183" candidate_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7" is_pre_election: true
I20260809 01:53:02.300009 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379f0cfaf959419b9a0f6855e600a183" candidate_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea5f5770434470d90b14f06c0884e53" is_pre_election: true
I20260809 01:53:02.300302 15120 raft_consensus.cc:2454] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0673839c2c44e13b2b9955b60f3f9d8 in term 0.
I20260809 01:53:02.300307 15317 raft_consensus.cc:2454] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0673839c2c44e13b2b9955b60f3f9d8 in term 0.
I20260809 01:53:02.300781 15424 leader_election.cc:302] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea5f5770434470d90b14f06c0884e53, c0673839c2c44e13b2b9955b60f3f9d8; no voters:
I20260809 01:53:02.301079 15619 raft_consensus.cc:2781] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:53:02.301162 15619 raft_consensus.cc:482] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:53:02.301205 15619 raft_consensus.cc:3037] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.302431 15619 raft_consensus.cc:504] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.302816 15619 leader_election.cc:288] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109), a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789)
I20260809 01:53:02.303184 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379f0cfaf959419b9a0f6855e600a183" candidate_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7"
I20260809 01:53:02.303190 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379f0cfaf959419b9a0f6855e600a183" candidate_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea5f5770434470d90b14f06c0884e53"
I20260809 01:53:02.303316 15120 raft_consensus.cc:3037] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.303323 15317 raft_consensus.cc:3037] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.304454 15317 raft_consensus.cc:2454] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0673839c2c44e13b2b9955b60f3f9d8 in term 1.
I20260809 01:53:02.304642 15120 raft_consensus.cc:2454] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0673839c2c44e13b2b9955b60f3f9d8 in term 1.
I20260809 01:53:02.304742 15425 leader_election.cc:302] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a024199de6b147fbac4a3f8292166fe7, c0673839c2c44e13b2b9955b60f3f9d8; no voters:
I20260809 01:53:02.304929 15619 raft_consensus.cc:2781] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:02.305190 15619 raft_consensus.cc:686] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 LEADER]: Becoming Leader. State: Replica: c0673839c2c44e13b2b9955b60f3f9d8, State: Running, Role: LEADER
I20260809 01:53:02.305567 15619 consensus_queue.cc:237] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } }
I20260809 01:53:02.308715 14883 catalog_manager.cc:5495] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3). New cstate: current_term: 1 leader_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } health_report { overall_health: UNKNOWN } } }
W20260809 01:53:02.311671 15574 tablet.cc:2369] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 01:53:02.325805 15388 tablet.cc:2369] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:53:02.332152 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.327952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.3.220.3:39969)
0809 01:53:02.328108 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.328113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.328136 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:53:02.328330 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.328334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.328447 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.328567 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:02.328877 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.329520 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.330545 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.330770 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.330853 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.331231 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:02.331697 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:02.331786 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.331927 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.331979 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:53:02.338260 15120 raft_consensus.cc:1261] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Refusing update from remote peer c0673839c2c44e13b2b9955b60f3f9d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:02.338264 15317 raft_consensus.cc:1261] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Refusing update from remote peer c0673839c2c44e13b2b9955b60f3f9d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:02.339046 15619 consensus_queue.cc:1035] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:02.339349 15623 consensus_queue.cc:1035] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:02.346343 15626 mvcc.cc:205] Tried to move back new op lower bound from 7316440606050476032 to 7316440605926514688. Current Snapshot: MvccSnapshot[applied={T|T < 7316440606050476032}]
I20260809 01:53:02.347726 15628 mvcc.cc:205] Tried to move back new op lower bound from 7316440606050476032 to 7316440605926514688. Current Snapshot: MvccSnapshot[applied={T|T < 7316440606050476032}]
I20260809 01:53:02.355170 14883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:02.355331 14883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:02.357829 14883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: Sending DeleteTablet for 3 replicas of tablet 379f0cfaf959419b9a0f6855e600a183
I20260809 01:53:02.358588 15091 tablet_service.cc:1514] Processing DeleteTablet for tablet 379f0cfaf959419b9a0f6855e600a183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:02 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:38218
I20260809 01:53:02.358808 15286 tablet_service.cc:1514] Processing DeleteTablet for tablet 379f0cfaf959419b9a0f6855e600a183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:02 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:50338
I20260809 01:53:02.358830 15492 tablet_service.cc:1514] Processing DeleteTablet for tablet 379f0cfaf959419b9a0f6855e600a183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:02 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:56718
I20260809 01:53:02.359237 15638 tablet_replica.cc:330] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: stopping tablet replica
I20260809 01:53:02.359453 15637 tablet_replica.cc:330] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: stopping tablet replica
I20260809 01:53:02.359665 15638 raft_consensus.cc:2229] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:02.359862 15637 raft_consensus.cc:2229] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:02.359827 14883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 01:53:02.360195 15638 raft_consensus.cc:2258] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 01:53:02.360219 14883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:53:02.360344 15637 raft_consensus.cc:2258] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:02.360939 15639 tablet_replica.cc:330] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: stopping tablet replica
I20260809 01:53:02.361284 15639 raft_consensus.cc:2229] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:53:02.361747 15639 raft_consensus.cc:2258] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:02.362160 15637 ts_tablet_manager.cc:1905] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:02.363099 15638 ts_tablet_manager.cc:1905] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:02.365235 15637 ts_tablet_manager.cc:1918] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:53:02.365448 15637 log.cc:1196] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/379f0cfaf959419b9a0f6855e600a183
I20260809 01:53:02.365522 15639 ts_tablet_manager.cc:1905] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:02.365919 15637 ts_tablet_manager.cc:1939] T 379f0cfaf959419b9a0f6855e600a183 P 6ea5f5770434470d90b14f06c0884e53: Deleting consensus metadata
I20260809 01:53:02.366219 15492 tablet_service.cc:1467] Processing CreateTablet for tablet 80f9d023f10043f5987f8518c69b3562 (DEFAULT_TABLE table=test-table [id=4c7af887feb34c1c9d3b659b78d54a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.366293 15638 ts_tablet_manager.cc:1918] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:53:02.366456 15638 log.cc:1196] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/379f0cfaf959419b9a0f6855e600a183
I20260809 01:53:02.366496 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f9d023f10043f5987f8518c69b3562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.366890 15638 ts_tablet_manager.cc:1939] T 379f0cfaf959419b9a0f6855e600a183 P a024199de6b147fbac4a3f8292166fe7: Deleting consensus metadata
I20260809 01:53:02.368233 14860 catalog_manager.cc:4841] TS a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789): tablet 379f0cfaf959419b9a0f6855e600a183 (table test-table [id=d98886cb05dd4869bb20edd4b5e8b24f]) successfully deleted
I20260809 01:53:02.368685 15611 tablet_bootstrap.cc:492] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: Bootstrap starting.
I20260809 01:53:02.368933 15639 ts_tablet_manager.cc:1918] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:53:02.369098 15639 log.cc:1196] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/379f0cfaf959419b9a0f6855e600a183
I20260809 01:53:02.369076 15286 tablet_service.cc:1467] Processing CreateTablet for tablet 80f9d023f10043f5987f8518c69b3562 (DEFAULT_TABLE table=test-table [id=4c7af887feb34c1c9d3b659b78d54a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.369343 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f9d023f10043f5987f8518c69b3562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.369498 15639 ts_tablet_manager.cc:1939] T 379f0cfaf959419b9a0f6855e600a183 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting consensus metadata
I20260809 01:53:02.369674 15611 tablet_bootstrap.cc:654] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.369951 14859 catalog_manager.cc:4841] TS 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109): tablet 379f0cfaf959419b9a0f6855e600a183 (table test-table [id=d98886cb05dd4869bb20edd4b5e8b24f]) successfully deleted
I20260809 01:53:02.370466 15091 tablet_service.cc:1467] Processing CreateTablet for tablet 80f9d023f10043f5987f8518c69b3562 (DEFAULT_TABLE table=test-table [id=4c7af887feb34c1c9d3b659b78d54a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.370754 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f9d023f10043f5987f8518c69b3562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.370924 14860 catalog_manager.cc:4841] TS c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969): tablet 379f0cfaf959419b9a0f6855e600a183 (table test-table [id=d98886cb05dd4869bb20edd4b5e8b24f]) successfully deleted
I20260809 01:53:02.372704 15610 tablet_bootstrap.cc:492] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: Bootstrap starting.
I20260809 01:53:02.373437 15612 tablet_bootstrap.cc:492] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: Bootstrap starting.
I20260809 01:53:02.373577 15610 tablet_bootstrap.cc:654] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.374405 15612 tablet_bootstrap.cc:654] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.374553 15611 tablet_bootstrap.cc:492] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: No bootstrap required, opened a new log
I20260809 01:53:02.374665 15611 ts_tablet_manager.cc:1397] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260809 01:53:02.375213 15611 raft_consensus.cc:348] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.375353 15611 raft_consensus.cc:374] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.375392 15611 raft_consensus.cc:729] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0673839c2c44e13b2b9955b60f3f9d8, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.375681 15611 consensus_queue.cc:260] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.375937 15612 tablet_bootstrap.cc:492] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: No bootstrap required, opened a new log
I20260809 01:53:02.376025 15612 ts_tablet_manager.cc:1397] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 01:53:02.376103 15611 ts_tablet_manager.cc:1428] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260809 01:53:02.376518 15612 raft_consensus.cc:348] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.376621 15612 raft_consensus.cc:374] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.376657 15612 raft_consensus.cc:729] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a024199de6b147fbac4a3f8292166fe7, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.376731 15610 tablet_bootstrap.cc:492] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: No bootstrap required, opened a new log
I20260809 01:53:02.376818 15610 ts_tablet_manager.cc:1397] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260809 01:53:02.376991 15612 consensus_queue.cc:260] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.377279 15610 raft_consensus.cc:348] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.378861 15610 raft_consensus.cc:374] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.378909 15610 raft_consensus.cc:729] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5f5770434470d90b14f06c0884e53, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.378966 15612 ts_tablet_manager.cc:1428] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260809 01:53:02.379029 15610 consensus_queue.cc:260] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.379273 15610 ts_tablet_manager.cc:1428] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 01:53:02.546085 15617 raft_consensus.cc:482] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:53:02.546281 15617 raft_consensus.cc:504] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.547389 15617 leader_election.cc:288] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789), c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969)
I20260809 01:53:02.551703 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.548295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:47147 (local address 127.3.220.3:39969)
0809 01:53:02.548465 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.548471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.548485 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:53:02.548602 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.548605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.548652 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.548732 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:02.549013 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.549507 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.550582 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.550995 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.551115 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.551190 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.551494 (+ 304us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.551573 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 01:53:02.551714 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.547789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:57771 (local address 127.3.220.2:46789)
0809 01:53:02.547946 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.547950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.548127 (+ 177us) server_negotiation.cc:407] Connection header received
0809 01:53:02.548299 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.548302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.548361 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.548476 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:02.548784 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.549306 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.550575 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.550991 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.551119 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.551187 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.551476 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.551559 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 01:53:02.551735 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.547702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:46789 (local address 127.3.220.1:57771)
0809 01:53:02.548031 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.548048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.548165 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.548507 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.548511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.548530 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:02.548656 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.548662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.549422 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.549427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.550444 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.550454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.551139 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.551143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.551330 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.551355 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.551442 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.551560 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":111,"thread_start_us":112,"threads_started":1}
I20260809 01:53:02.551736 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.548109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:39969 (local address 127.3.220.1:47147)
0809 01:53:02.548390 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.548404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.548486 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.548755 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.548759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.548774 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:02.548905 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.548910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.549659 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.549662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.550455 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.550463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.551142 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.551146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.551348 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.551368 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.551470 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.551579 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":74,"thread_start_us":103,"threads_started":1}
I20260809 01:53:02.552405 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f9d023f10043f5987f8518c69b3562" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7" is_pre_election: true
I20260809 01:53:02.552435 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f9d023f10043f5987f8518c69b3562" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" is_pre_election: true
I20260809 01:53:02.552569 15317 raft_consensus.cc:2454] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 0.
I20260809 01:53:02.552721 15521 raft_consensus.cc:2454] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 0.
I20260809 01:53:02.552975 15020 leader_election.cc:302] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea5f5770434470d90b14f06c0884e53, a024199de6b147fbac4a3f8292166fe7; no voters:
I20260809 01:53:02.553306 15617 raft_consensus.cc:2781] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:53:02.553375 15617 raft_consensus.cc:482] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:53:02.553412 15617 raft_consensus.cc:3037] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.554463 15617 raft_consensus.cc:504] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.554813 15617 leader_election.cc:288] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 election: Requested vote from peers a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789), c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969)
I20260809 01:53:02.555207 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f9d023f10043f5987f8518c69b3562" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0673839c2c44e13b2b9955b60f3f9d8"
I20260809 01:53:02.555197 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f9d023f10043f5987f8518c69b3562" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7"
I20260809 01:53:02.555327 15521 raft_consensus.cc:3037] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.555342 15317 raft_consensus.cc:3037] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.556496 15317 raft_consensus.cc:2454] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 1.
I20260809 01:53:02.556573 15521 raft_consensus.cc:2454] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 1.
I20260809 01:53:02.556805 15020 leader_election.cc:302] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea5f5770434470d90b14f06c0884e53, a024199de6b147fbac4a3f8292166fe7; no voters:
I20260809 01:53:02.556991 15617 raft_consensus.cc:2781] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:02.557207 15617 raft_consensus.cc:686] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea5f5770434470d90b14f06c0884e53, State: Running, Role: LEADER
I20260809 01:53:02.557504 15617 consensus_queue.cc:237] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.559867 14879 catalog_manager.cc:5495] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } health_report { overall_health: UNKNOWN } } }
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240381, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.62@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
W20260809 01:53:02.601956 15186 tablet.cc:2369] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:53:02.604833 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.595536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.3.220.62:42101)
0809 01:53:02.595665 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.595669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.595913 (+ 244us) server_negotiation.cc:407] Connection header received
0809 01:53:02.596027 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.596029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.596074 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.596145 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:02.596853 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.597374 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.598310 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.598613 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.598721 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.601012 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:02.601016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:02.601019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:02.601043 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:02.602801 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.603288 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.603300 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.603301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.603393 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.603807 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.603810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.603811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.604051 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:02.604165 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.604508 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.604646 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:53:02.604868 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.595512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42101 (local address 127.0.0.1:54436)
0809 01:53:02.595816 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.595828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.595941 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.596216 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.596218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.596626 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:02.596742 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.596748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.597504 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.597507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.598173 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.598182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.598742 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.598744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.598826 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.599443 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:02.599454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:02.600869 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:02.602909 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:02.602918 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:02.602923 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:02.603162 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:02.603555 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:02.603559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:02.603562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:02.603700 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:02.604203 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:02.604210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:02.604330 (+ 120us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.604434 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.604556 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":142,"threads_started":1}
I20260809 01:53:02.608929 14879 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:54436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 01:53:02.609817 14879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:53:02.616616 15091 tablet_service.cc:1467] Processing CreateTablet for tablet 034eb26261df451aa0d597833bffe89f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d42aede32f75437eb0e79cffd093f79d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:53:02.617053 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034eb26261df451aa0d597833bffe89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.617362 15492 tablet_service.cc:1467] Processing CreateTablet for tablet 034eb26261df451aa0d597833bffe89f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d42aede32f75437eb0e79cffd093f79d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:53:02.617587 15286 tablet_service.cc:1467] Processing CreateTablet for tablet 034eb26261df451aa0d597833bffe89f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d42aede32f75437eb0e79cffd093f79d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 01:53:02.617959 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034eb26261df451aa0d597833bffe89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.618374 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034eb26261df451aa0d597833bffe89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.620335 15610 tablet_bootstrap.cc:492] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: Bootstrap starting.
I20260809 01:53:02.621251 15610 tablet_bootstrap.cc:654] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.624396 15612 tablet_bootstrap.cc:492] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: Bootstrap starting.
I20260809 01:53:02.625182 15610 tablet_bootstrap.cc:492] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: No bootstrap required, opened a new log
I20260809 01:53:02.625281 15610 ts_tablet_manager.cc:1397] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:53:02.625427 15612 tablet_bootstrap.cc:654] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.625962 15610 raft_consensus.cc:348] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.626111 15610 raft_consensus.cc:374] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.626178 15610 raft_consensus.cc:729] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5f5770434470d90b14f06c0884e53, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.626268 15611 tablet_bootstrap.cc:492] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: Bootstrap starting.
I20260809 01:53:02.626332 15610 consensus_queue.cc:260] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.626591 15617 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.626765 15617 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:53:02.626878 15610 ts_tablet_manager.cc:1428] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 01:53:02.627329 15611 tablet_bootstrap.cc:654] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.628310 15612 tablet_bootstrap.cc:492] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: No bootstrap required, opened a new log
I20260809 01:53:02.628449 15612 ts_tablet_manager.cc:1397] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260809 01:53:02.628708 15611 tablet_bootstrap.cc:492] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: No bootstrap required, opened a new log
I20260809 01:53:02.628813 15611 ts_tablet_manager.cc:1397] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 01:53:02.628938 15612 raft_consensus.cc:348] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.629065 15612 raft_consensus.cc:374] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.629103 15612 raft_consensus.cc:729] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a024199de6b147fbac4a3f8292166fe7, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.629248 15612 consensus_queue.cc:260] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.629333 15611 raft_consensus.cc:348] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.629470 15611 raft_consensus.cc:374] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.629519 15611 raft_consensus.cc:729] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0673839c2c44e13b2b9955b60f3f9d8, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.629686 15611 consensus_queue.cc:260] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.629765 15618 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.629942 15618 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:53:02.629997 15611 ts_tablet_manager.cc:1428] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260809 01:53:02.629774 15612 ts_tablet_manager.cc:1428] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 01:53:02.630241 15619 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.630380 15619 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:53:02.730399 15617 raft_consensus.cc:482] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:53:02.730553 15617 raft_consensus.cc:504] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.730970 15617 leader_election.cc:288] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789), c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969)
I20260809 01:53:02.731611 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034eb26261df451aa0d597833bffe89f" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7" is_pre_election: true
I20260809 01:53:02.731611 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034eb26261df451aa0d597833bffe89f" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" is_pre_election: true
I20260809 01:53:02.731793 15317 raft_consensus.cc:2454] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 0.
I20260809 01:53:02.731793 15521 raft_consensus.cc:2454] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 0.
I20260809 01:53:02.732117 15020 leader_election.cc:302] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea5f5770434470d90b14f06c0884e53, a024199de6b147fbac4a3f8292166fe7; no voters:
I20260809 01:53:02.732348 15617 raft_consensus.cc:2781] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:53:02.732420 15617 raft_consensus.cc:482] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:53:02.732466 15617 raft_consensus.cc:3037] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.733587 15617 raft_consensus.cc:504] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.733956 15617 leader_election.cc:288] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 election: Requested vote from peers a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789), c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969)
I20260809 01:53:02.734340 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034eb26261df451aa0d597833bffe89f" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7"
I20260809 01:53:02.734476 15317 raft_consensus.cc:3037] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.734472 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034eb26261df451aa0d597833bffe89f" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0673839c2c44e13b2b9955b60f3f9d8"
I20260809 01:53:02.734627 15521 raft_consensus.cc:3037] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.735607 15317 raft_consensus.cc:2454] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 1.
I20260809 01:53:02.735766 15521 raft_consensus.cc:2454] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 1.
I20260809 01:53:02.735954 15020 leader_election.cc:302] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea5f5770434470d90b14f06c0884e53, a024199de6b147fbac4a3f8292166fe7; no voters:
I20260809 01:53:02.736147 15617 raft_consensus.cc:2781] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:02.736249 15617 raft_consensus.cc:686] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea5f5770434470d90b14f06c0884e53, State: Running, Role: LEADER
I20260809 01:53:02.736402 15617 consensus_queue.cc:237] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.737212 15649 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: TxnStatusTablet state changed. Reason: New leader 6ea5f5770434470d90b14f06c0884e53. Latest consensus state: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.737335 15649 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:53:02.737708 15669 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 01:53:02.737895 15669 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 01:53:02.738368 14883 catalog_manager.cc:5495] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } health_report { overall_health: UNKNOWN } } }
I20260809 01:53:02.756134 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.751994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:44109 (local address 127.0.0.1:38220)
0809 01:53:02.752158 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.752186 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.752273 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.752614 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.752616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.752625 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:53:02.752734 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.752738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.753815 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.753818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.754441 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.754449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.755020 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.755022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.755086 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.755131 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:53:02.755885 (+ 754us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:53:02.755898 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.755950 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.756018 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 01:53:02.756152 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.752109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.3.220.1:44109)
0809 01:53:02.752234 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.752239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.752254 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:53:02.752417 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.752421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.752486 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.752596 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:02.752840 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.753682 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.754552 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.754908 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.755010 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.755266 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:02.755777 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:02.755898 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.755964 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.756021 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 01:53:02.759052 15317 raft_consensus.cc:1261] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Refusing update from remote peer 6ea5f5770434470d90b14f06c0884e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:02.759092 15521 raft_consensus.cc:1261] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Refusing update from remote peer 6ea5f5770434470d90b14f06c0884e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:02.759552 15649 consensus_queue.cc:1035] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:02.759743 15617 consensus_queue.cc:1035] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:02.761265 15618 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: TxnStatusTablet state changed. Reason: New leader 6ea5f5770434470d90b14f06c0884e53. Latest consensus state: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.761379 15618 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:53:02.761579 15619 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: TxnStatusTablet state changed. Reason: New leader 6ea5f5770434470d90b14f06c0884e53. Latest consensus state: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.761704 15619 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:53:02.763919 15676 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.764073 15676 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:53:02.764324 15618 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.764441 15618 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P a024199de6b147fbac4a3f8292166fe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:53:02.766974 15619 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.767081 15619 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P c0673839c2c44e13b2b9955b60f3f9d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 01:53:02.768484 15617 tablet_replica.cc:439] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } } }
I20260809 01:53:02.768585 15617 tablet_replica.cc:442] T 034eb26261df451aa0d597833bffe89f P 6ea5f5770434470d90b14f06c0884e53: This TxnStatusTablet replica's current role is: LEADER
I20260809 01:53:02.774633 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.770979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.3.220.1:44109)
0809 01:53:02.771337 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.771341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.771353 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:53:02.771392 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.771395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.771438 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.771512 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:02.771788 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.772557 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.773371 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.773744 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.773902 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.774171 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:02.774248 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:02.774320 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.774451 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.774503 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260809 01:53:02.785373 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.781725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:54071 (local address 127.3.220.1:44109)
0809 01:53:02.781853 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.781856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.782102 (+ 246us) server_negotiation.cc:407] Connection header received
0809 01:53:02.782204 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.782206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.782246 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.782309 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:02.782586 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.783046 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.784317 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.784517 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.784592 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.784943 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:02.785003 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:02.785055 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.785193 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.785265 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":47}
I20260809 01:53:02.785450 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.781638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:44109 (local address 127.3.220.1:54071)
0809 01:53:02.781990 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:02.782004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:02.782105 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:02.782355 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:02.782358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:02.782366 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:53:02.782478 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.782484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.783148 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.783151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.784192 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:02.784203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.784630 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.784633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:02.784756 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.784816 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:53:02.785090 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:53:02.785097 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 01:53:02.785163 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 01:53:02.785257 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260809 01:53:02.792229 15521 raft_consensus.cc:1261] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Refusing update from remote peer 6ea5f5770434470d90b14f06c0884e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:02.792236 15317 raft_consensus.cc:1261] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Refusing update from remote peer 6ea5f5770434470d90b14f06c0884e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:02.792616 15649 consensus_queue.cc:1035] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:02.792783 15676 consensus_queue.cc:1035] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:02.796729 15627 mvcc.cc:205] Tried to move back new op lower bound from 7316440607911223296 to 7316440606958149632. Current Snapshot: MvccSnapshot[applied={T|T < 7316440607911223296}]
I20260809 01:53:02.835736 14883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:02.835917 14883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:02.838268 14883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: Sending DeleteTablet for 3 replicas of tablet 80f9d023f10043f5987f8518c69b3562
I20260809 01:53:02.838812 15090 tablet_service.cc:1514] Processing DeleteTablet for tablet 80f9d023f10043f5987f8518c69b3562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:02 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:38218
I20260809 01:53:02.838826 15286 tablet_service.cc:1514] Processing DeleteTablet for tablet 80f9d023f10043f5987f8518c69b3562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:02 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:50338
I20260809 01:53:02.839012 15638 tablet_replica.cc:330] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: stopping tablet replica
I20260809 01:53:02.839015 15637 tablet_replica.cc:330] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: stopping tablet replica
I20260809 01:53:02.839061 15492 tablet_service.cc:1514] Processing DeleteTablet for tablet 80f9d023f10043f5987f8518c69b3562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:02 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:56718
I20260809 01:53:02.839123 15638 raft_consensus.cc:2229] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:02.839123 15637 raft_consensus.cc:2229] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:53:02.839236 15638 raft_consensus.cc:2258] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:02.839265 15639 tablet_replica.cc:330] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: stopping tablet replica
I20260809 01:53:02.839382 15639 raft_consensus.cc:2229] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:02.839442 15637 raft_consensus.cc:2258] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:02.839506 15639 raft_consensus.cc:2258] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:02.840163 15637 ts_tablet_manager.cc:1905] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:02.840409 15639 ts_tablet_manager.cc:1905] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:02.842859 15639 ts_tablet_manager.cc:1918] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:53:02.842940 15639 log.cc:1196] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/80f9d023f10043f5987f8518c69b3562
I20260809 01:53:02.842926 15637 ts_tablet_manager.cc:1918] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:53:02.842990 15637 log.cc:1196] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/80f9d023f10043f5987f8518c69b3562
I20260809 01:53:02.843228 15639 ts_tablet_manager.cc:1939] T 80f9d023f10043f5987f8518c69b3562 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting consensus metadata
I20260809 01:53:02.843267 15637 ts_tablet_manager.cc:1939] T 80f9d023f10043f5987f8518c69b3562 P 6ea5f5770434470d90b14f06c0884e53: Deleting consensus metadata
I20260809 01:53:02.843932 14859 catalog_manager.cc:4841] TS 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109): tablet 80f9d023f10043f5987f8518c69b3562 (table test-table [id=4c7af887feb34c1c9d3b659b78d54a5e]) successfully deleted
I20260809 01:53:02.844314 14860 catalog_manager.cc:4841] TS c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969): tablet 80f9d023f10043f5987f8518c69b3562 (table test-table [id=4c7af887feb34c1c9d3b659b78d54a5e]) successfully deleted
I20260809 01:53:02.848847 15638 ts_tablet_manager.cc:1905] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:02.851432 15638 ts_tablet_manager.cc:1918] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:53:02.851536 15638 log.cc:1196] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/80f9d023f10043f5987f8518c69b3562
I20260809 01:53:02.851775 15638 ts_tablet_manager.cc:1939] T 80f9d023f10043f5987f8518c69b3562 P a024199de6b147fbac4a3f8292166fe7: Deleting consensus metadata
I20260809 01:53:02.852448 14860 catalog_manager.cc:4841] TS a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789): tablet 80f9d023f10043f5987f8518c69b3562 (table test-table [id=4c7af887feb34c1c9d3b659b78d54a5e]) successfully deleted
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 01:53:02.874270 14963 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:02.866574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.3.220.62:42101)
0809 01:53:02.866738 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.866744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.866927 (+ 183us) server_negotiation.cc:407] Connection header received
0809 01:53:02.867070 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.867072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.867120 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.867219 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:02.868124 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.868676 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.870005 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.870260 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.870361 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.874101 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54448: BlockingRecv error: recv got EOF from 127.0.0.1:54448 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 01:53:02 dist-test-slave-gkw7 krb5kdc[14821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM
I20260809 01:53:02.885058 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:02.875978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.3.220.62:42101)
0809 01:53:02.876094 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:02.876099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:02.876345 (+ 246us) server_negotiation.cc:407] Connection header received
0809 01:53:02.876509 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:02.876511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:02.876553 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:02.876637 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:02.877252 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.877828 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.878607 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:02.878826 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:02.878939 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:02.881469 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:02.881473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:02.881481 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:02.881509 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:02.883285 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.883755 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.883758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.883760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.883863 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:02.884139 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:02.884142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:02.884144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:02.884372 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:02.884515 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:02.884746 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 01:53:02.884881 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:53:02.888120 14883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:53:02.888630 14883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:53:02.894022 15090 tablet_service.cc:1467] Processing CreateTablet for tablet ccf2eb5f59c847f2a6be5e12630484d8 (DEFAULT_TABLE table=test-table [id=5f73cbfb95444d56841e58ea5cd7ccab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.894068 15286 tablet_service.cc:1467] Processing CreateTablet for tablet ccf2eb5f59c847f2a6be5e12630484d8 (DEFAULT_TABLE table=test-table [id=5f73cbfb95444d56841e58ea5cd7ccab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.894290 15492 tablet_service.cc:1467] Processing CreateTablet for tablet ccf2eb5f59c847f2a6be5e12630484d8 (DEFAULT_TABLE table=test-table [id=5f73cbfb95444d56841e58ea5cd7ccab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:02.894404 15090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf2eb5f59c847f2a6be5e12630484d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.894402 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf2eb5f59c847f2a6be5e12630484d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.894554 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf2eb5f59c847f2a6be5e12630484d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:02.897013 15610 tablet_bootstrap.cc:492] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: Bootstrap starting.
I20260809 01:53:02.897094 15611 tablet_bootstrap.cc:492] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: Bootstrap starting.
I20260809 01:53:02.897974 15611 tablet_bootstrap.cc:654] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.898089 15610 tablet_bootstrap.cc:654] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.899111 15612 tablet_bootstrap.cc:492] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: Bootstrap starting.
I20260809 01:53:02.899179 15611 tablet_bootstrap.cc:492] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: No bootstrap required, opened a new log
I20260809 01:53:02.899250 15611 ts_tablet_manager.cc:1397] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 01:53:02.899654 15611 raft_consensus.cc:348] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.899778 15611 raft_consensus.cc:374] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.899775 15610 tablet_bootstrap.cc:492] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: No bootstrap required, opened a new log
I20260809 01:53:02.899812 15611 raft_consensus.cc:729] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0673839c2c44e13b2b9955b60f3f9d8, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.899876 15610 ts_tablet_manager.cc:1397] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 01:53:02.899943 15611 consensus_queue.cc:260] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.900044 15612 tablet_bootstrap.cc:654] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:02.900266 15611 ts_tablet_manager.cc:1428] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 01:53:02.900310 15610 raft_consensus.cc:348] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.900417 15610 raft_consensus.cc:374] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.900453 15610 raft_consensus.cc:729] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5f5770434470d90b14f06c0884e53, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.900579 15610 consensus_queue.cc:260] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.900842 15610 ts_tablet_manager.cc:1428] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 01:53:02.903556 15612 tablet_bootstrap.cc:492] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: No bootstrap required, opened a new log
I20260809 01:53:02.903676 15612 ts_tablet_manager.cc:1397] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260809 01:53:02.904335 15612 raft_consensus.cc:348] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.904467 15612 raft_consensus.cc:374] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:02.904505 15612 raft_consensus.cc:729] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a024199de6b147fbac4a3f8292166fe7, State: Initialized, Role: FOLLOWER
I20260809 01:53:02.904642 15612 consensus_queue.cc:260] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.905362 15612 ts_tablet_manager.cc:1428] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 01:53:02.987458 15676 raft_consensus.cc:482] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:53:02.987634 15676 raft_consensus.cc:504] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.988102 15676 leader_election.cc:288] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789), c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969)
I20260809 01:53:02.988766 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf2eb5f59c847f2a6be5e12630484d8" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" is_pre_election: true
I20260809 01:53:02.988758 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf2eb5f59c847f2a6be5e12630484d8" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7" is_pre_election: true
I20260809 01:53:02.988965 15521 raft_consensus.cc:2454] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 0.
I20260809 01:53:02.988965 15312 raft_consensus.cc:2454] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 0.
I20260809 01:53:02.989332 15020 leader_election.cc:302] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea5f5770434470d90b14f06c0884e53, c0673839c2c44e13b2b9955b60f3f9d8; no voters:
I20260809 01:53:02.989637 15676 raft_consensus.cc:2781] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:53:02.989729 15676 raft_consensus.cc:482] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:53:02.989773 15676 raft_consensus.cc:3037] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.991089 15676 raft_consensus.cc:504] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.991441 15676 leader_election.cc:288] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 election: Requested vote from peers a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789), c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969)
I20260809 01:53:02.991891 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf2eb5f59c847f2a6be5e12630484d8" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0673839c2c44e13b2b9955b60f3f9d8"
I20260809 01:53:02.991905 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf2eb5f59c847f2a6be5e12630484d8" candidate_uuid: "6ea5f5770434470d90b14f06c0884e53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7"
I20260809 01:53:02.992034 15312 raft_consensus.cc:3037] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.992033 15521 raft_consensus.cc:3037] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:02.993072 15521 raft_consensus.cc:2454] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 1.
I20260809 01:53:02.993285 15312 raft_consensus.cc:2454] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea5f5770434470d90b14f06c0884e53 in term 1.
I20260809 01:53:02.993448 15020 leader_election.cc:302] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea5f5770434470d90b14f06c0884e53, c0673839c2c44e13b2b9955b60f3f9d8; no voters:
I20260809 01:53:02.993659 15676 raft_consensus.cc:2781] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:02.993775 15676 raft_consensus.cc:686] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea5f5770434470d90b14f06c0884e53, State: Running, Role: LEADER
I20260809 01:53:02.993933 15676 consensus_queue.cc:237] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:02.995791 14879 catalog_manager.cc:5495] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "6ea5f5770434470d90b14f06c0884e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } health_report { overall_health: UNKNOWN } } }
I20260809 01:53:03.007797 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:03.004173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.3.220.1:44109)
0809 01:53:03.004315 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:03.004320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:03.004335 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:53:03.004384 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:03.004387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:03.004438 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:03.004523 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:03.004789 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.005600 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.006598 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.006950 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.007056 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:03.007300 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:03.007385 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:03.007469 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:03.007578 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 01:53:03.007630 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:53:03.009908 15312 raft_consensus.cc:1261] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Refusing update from remote peer 6ea5f5770434470d90b14f06c0884e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:03.009935 15521 raft_consensus.cc:1261] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Refusing update from remote peer 6ea5f5770434470d90b14f06c0884e53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:03.010367 15676 consensus_queue.cc:1035] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:03.010540 15617 consensus_queue.cc:1035] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:03.020361 14883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:03.020529 14883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:03.022643 14883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: Sending DeleteTablet for 3 replicas of tablet ccf2eb5f59c847f2a6be5e12630484d8
I20260809 01:53:03.023232 15090 tablet_service.cc:1514] Processing DeleteTablet for tablet ccf2eb5f59c847f2a6be5e12630484d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:03 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:38218
I20260809 01:53:03.023254 15492 tablet_service.cc:1514] Processing DeleteTablet for tablet ccf2eb5f59c847f2a6be5e12630484d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:03 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:56718
I20260809 01:53:03.023442 15637 tablet_replica.cc:330] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: stopping tablet replica
I20260809 01:53:03.023445 15286 tablet_service.cc:1514] Processing DeleteTablet for tablet ccf2eb5f59c847f2a6be5e12630484d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:03 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:50338
I20260809 01:53:03.023566 15637 raft_consensus.cc:2229] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:53:03.023619 15638 tablet_replica.cc:330] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: stopping tablet replica
I20260809 01:53:03.023633 15639 tablet_replica.cc:330] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: stopping tablet replica
I20260809 01:53:03.023732 15638 raft_consensus.cc:2229] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:03.023764 15639 raft_consensus.cc:2229] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:03.023773 15637 raft_consensus.cc:2258] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:03.023849 15638 raft_consensus.cc:2258] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:03.023886 15639 raft_consensus.cc:2258] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:03.024470 15638 ts_tablet_manager.cc:1905] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:03.024470 15637 ts_tablet_manager.cc:1905] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:03.025313 15639 ts_tablet_manager.cc:1905] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:03.026933 15637 ts_tablet_manager.cc:1918] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:53:03.026933 15638 ts_tablet_manager.cc:1918] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:53:03.027006 15637 log.cc:1196] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/ccf2eb5f59c847f2a6be5e12630484d8
I20260809 01:53:03.027006 15638 log.cc:1196] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/ccf2eb5f59c847f2a6be5e12630484d8
I20260809 01:53:03.027278 15638 ts_tablet_manager.cc:1939] T ccf2eb5f59c847f2a6be5e12630484d8 P a024199de6b147fbac4a3f8292166fe7: Deleting consensus metadata
I20260809 01:53:03.027285 15637 ts_tablet_manager.cc:1939] T ccf2eb5f59c847f2a6be5e12630484d8 P 6ea5f5770434470d90b14f06c0884e53: Deleting consensus metadata
I20260809 01:53:03.028015 14860 catalog_manager.cc:4841] TS a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789): tablet ccf2eb5f59c847f2a6be5e12630484d8 (table test-table [id=5f73cbfb95444d56841e58ea5cd7ccab]) successfully deleted
I20260809 01:53:03.028015 14859 catalog_manager.cc:4841] TS 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109): tablet ccf2eb5f59c847f2a6be5e12630484d8 (table test-table [id=5f73cbfb95444d56841e58ea5cd7ccab]) successfully deleted
I20260809 01:53:03.028352 15639 ts_tablet_manager.cc:1918] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:53:03.028420 15639 log.cc:1196] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/ccf2eb5f59c847f2a6be5e12630484d8
I20260809 01:53:03.028952 15639 ts_tablet_manager.cc:1939] T ccf2eb5f59c847f2a6be5e12630484d8 P c0673839c2c44e13b2b9955b60f3f9d8: Deleting consensus metadata
I20260809 01:53:03.029724 14860 catalog_manager.cc:4841] TS c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969): tablet ccf2eb5f59c847f2a6be5e12630484d8 (table test-table [id=5f73cbfb95444d56841e58ea5cd7ccab]) successfully deleted
I20260809 01:53:03.030148 14883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:53:03.030596 14883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:53:03.035260 15090 tablet_service.cc:1467] Processing CreateTablet for tablet dfbab53279ef4f54ba7964d61391edab (DEFAULT_TABLE table=test-table [id=a2586d75e93e4f70a4d5eaa18a20b0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:03.035559 15090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbab53279ef4f54ba7964d61391edab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:03.035524 15286 tablet_service.cc:1467] Processing CreateTablet for tablet dfbab53279ef4f54ba7964d61391edab (DEFAULT_TABLE table=test-table [id=a2586d75e93e4f70a4d5eaa18a20b0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:03.035789 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbab53279ef4f54ba7964d61391edab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:03.035670 15492 tablet_service.cc:1467] Processing CreateTablet for tablet dfbab53279ef4f54ba7964d61391edab (DEFAULT_TABLE table=test-table [id=a2586d75e93e4f70a4d5eaa18a20b0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:03.036027 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbab53279ef4f54ba7964d61391edab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:03.038182 15612 tablet_bootstrap.cc:492] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: Bootstrap starting.
I20260809 01:53:03.038304 15610 tablet_bootstrap.cc:492] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53: Bootstrap starting.
I20260809 01:53:03.038357 15611 tablet_bootstrap.cc:492] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: Bootstrap starting.
I20260809 01:53:03.039120 15610 tablet_bootstrap.cc:654] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:03.039120 15611 tablet_bootstrap.cc:654] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:03.039134 15612 tablet_bootstrap.cc:654] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:03.040570 15612 tablet_bootstrap.cc:492] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: No bootstrap required, opened a new log
I20260809 01:53:03.040729 15612 ts_tablet_manager.cc:1397] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260809 01:53:03.040802 15611 tablet_bootstrap.cc:492] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: No bootstrap required, opened a new log
I20260809 01:53:03.040892 15611 ts_tablet_manager.cc:1397] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 01:53:03.041018 15610 tablet_bootstrap.cc:492] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53: No bootstrap required, opened a new log
I20260809 01:53:03.041090 15610 ts_tablet_manager.cc:1397] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 01:53:03.041193 15612 raft_consensus.cc:348] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.041301 15612 raft_consensus.cc:374] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:03.041329 15612 raft_consensus.cc:729] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a024199de6b147fbac4a3f8292166fe7, State: Initialized, Role: FOLLOWER
I20260809 01:53:03.041296 15611 raft_consensus.cc:348] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.041399 15611 raft_consensus.cc:374] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:03.041433 15611 raft_consensus.cc:729] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0673839c2c44e13b2b9955b60f3f9d8, State: Initialized, Role: FOLLOWER
I20260809 01:53:03.041473 15612 consensus_queue.cc:260] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.041479 15610 raft_consensus.cc:348] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.041612 15610 raft_consensus.cc:374] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:03.041584 15611 consensus_queue.cc:260] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.041646 15610 raft_consensus.cc:729] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5f5770434470d90b14f06c0884e53, State: Initialized, Role: FOLLOWER
I20260809 01:53:03.041762 15610 consensus_queue.cc:260] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.041905 15611 ts_tablet_manager.cc:1428] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:53:03.042042 15610 ts_tablet_manager.cc:1428] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:53:03.043635 15612 ts_tablet_manager.cc:1428] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260809 01:53:03.106388 15619 raft_consensus.cc:482] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:53:03.106534 15619 raft_consensus.cc:504] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.106959 15619 leader_election.cc:288] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109), a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789)
I20260809 01:53:03.107477 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbab53279ef4f54ba7964d61391edab" candidate_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea5f5770434470d90b14f06c0884e53" is_pre_election: true
I20260809 01:53:03.107471 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbab53279ef4f54ba7964d61391edab" candidate_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7" is_pre_election: true
I20260809 01:53:03.107663 15120 raft_consensus.cc:2454] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0673839c2c44e13b2b9955b60f3f9d8 in term 0.
I20260809 01:53:03.107659 15312 raft_consensus.cc:2454] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0673839c2c44e13b2b9955b60f3f9d8 in term 0.
I20260809 01:53:03.107944 15425 leader_election.cc:302] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a024199de6b147fbac4a3f8292166fe7, c0673839c2c44e13b2b9955b60f3f9d8; no voters:
I20260809 01:53:03.108104 15619 raft_consensus.cc:2781] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:53:03.108161 15619 raft_consensus.cc:482] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:53:03.108188 15619 raft_consensus.cc:3037] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:03.109207 15619 raft_consensus.cc:504] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.109596 15619 leader_election.cc:288] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109), a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789)
I20260809 01:53:03.109969 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbab53279ef4f54ba7964d61391edab" candidate_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a024199de6b147fbac4a3f8292166fe7"
I20260809 01:53:03.109982 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbab53279ef4f54ba7964d61391edab" candidate_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea5f5770434470d90b14f06c0884e53"
I20260809 01:53:03.110109 15120 raft_consensus.cc:3037] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:03.110097 15312 raft_consensus.cc:3037] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:03.111060 15120 raft_consensus.cc:2454] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0673839c2c44e13b2b9955b60f3f9d8 in term 1.
I20260809 01:53:03.111141 15312 raft_consensus.cc:2454] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0673839c2c44e13b2b9955b60f3f9d8 in term 1.
I20260809 01:53:03.111344 15424 leader_election.cc:302] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea5f5770434470d90b14f06c0884e53, c0673839c2c44e13b2b9955b60f3f9d8; no voters:
I20260809 01:53:03.111533 15619 raft_consensus.cc:2781] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:03.111730 15619 raft_consensus.cc:686] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 LEADER]: Becoming Leader. State: Replica: c0673839c2c44e13b2b9955b60f3f9d8, State: Running, Role: LEADER
I20260809 01:53:03.111879 15619 consensus_queue.cc:237] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } }
I20260809 01:53:03.113565 14883 catalog_manager.cc:5495] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3). New cstate: current_term: 1 leader_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0673839c2c44e13b2b9955b60f3f9d8" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39969 } health_report { overall_health: HEALTHY } } }
I20260809 01:53:03.136034 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:03.132812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.3.220.3:39969)
0809 01:53:03.133072 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:03.133077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:03.133095 (+ 18us) server_negotiation.cc:407] Connection header received
0809 01:53:03.133137 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:03.133140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:03.133179 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:03.133272 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:03.133558 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.134087 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.134946 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.135173 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.135278 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:03.135547 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:03.135618 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:03.135702 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:03.135828 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 01:53:03.135881 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260809 01:53:03.145462 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:03.142486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:51343 (local address 127.3.220.1:44109)
0809 01:53:03.142613 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:03.142616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:03.142773 (+ 157us) server_negotiation.cc:407] Connection header received
0809 01:53:03.142879 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:03.142881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:03.142923 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:03.143003 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:03.143190 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.143686 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.144513 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.144731 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.144810 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:03.145045 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:03.145108 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:03.145170 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:03.145314 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 01:53:03.145357 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:53:03.145494 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:03.142401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:44109 (local address 127.3.220.3:51343)
0809 01:53:03.142682 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:03.142703 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:03.142798 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:03.143011 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:03.143013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:03.143019 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:53:03.143099 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:03.143103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.143785 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.143787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:03.144411 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:03.144420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.144826 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.144828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:03.144892 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:03.144932 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:53:03.145213 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:53:03.145220 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 01:53:03.145298 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 01:53:03.145381 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
I20260809 01:53:03.151450 15312 raft_consensus.cc:1261] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Refusing update from remote peer c0673839c2c44e13b2b9955b60f3f9d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:03.151562 15120 raft_consensus.cc:1261] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Refusing update from remote peer c0673839c2c44e13b2b9955b60f3f9d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:03.151937 15619 consensus_queue.cc:1035] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a024199de6b147fbac4a3f8292166fe7" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:03.152088 15715 consensus_queue.cc:1035] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea5f5770434470d90b14f06c0884e53" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:03.169116 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:03.165891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:34161 (local address 127.3.220.3:39969)
0809 01:53:03.166031 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:03.166035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:03.166052 (+ 17us) server_negotiation.cc:407] Connection header received
0809 01:53:03.166189 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:03.166194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:03.166257 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:03.166334 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:03.166576 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.167108 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.167946 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.168249 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.168352 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:03.168621 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:03.168709 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:03.168787 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:03.168928 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 01:53:03.168993 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 01:53:03.169118 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:03.165798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:39969 (local address 127.3.220.1:34161)
0809 01:53:03.165999 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:03.166014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:03.166082 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:03.166354 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:03.166357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:03.166365 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 01:53:03.166476 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:03.166483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.167207 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.167210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:03.167848 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:03.167856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:03.168357 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:03.168361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:03.168455 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:03.168497 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 01:53:03.168814 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 01:53:03.168823 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 01:53:03.168896 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 01:53:03.168969 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260809 01:53:03.188581 14883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:03.188736 14883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:03.190682 14883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 569dd80c966042ea9a9ebf5caeee32a0: Sending DeleteTablet for 3 replicas of tablet dfbab53279ef4f54ba7964d61391edab
I20260809 01:53:03.191139 15090 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbab53279ef4f54ba7964d61391edab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:03 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:38218
I20260809 01:53:03.191315 15286 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbab53279ef4f54ba7964d61391edab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:03 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:50338
I20260809 01:53:03.191340 15637 tablet_replica.cc:330] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53: stopping tablet replica
I20260809 01:53:03.191504 15638 tablet_replica.cc:330] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: stopping tablet replica
I20260809 01:53:03.191488 15637 raft_consensus.cc:2229] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:03.191618 15637 raft_consensus.cc:2258] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:03.191640 15638 raft_consensus.cc:2229] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:03.191808 15638 raft_consensus.cc:2258] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:03.192051 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 14995
I20260809 01:53:03.192287 15637 ts_tablet_manager.cc:1905] T dfbab53279ef4f54ba7964d61391edab P 6ea5f5770434470d90b14f06c0884e53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:03.196640 15492 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbab53279ef4f54ba7964d61391edab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:03 UTC) from {username='oryx', principal='oryx/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:56718
I20260809 01:53:03.196846 15639 tablet_replica.cc:330] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: stopping tablet replica
I20260809 01:53:03.197001 15639 raft_consensus.cc:2229] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:53:03.197268 15639 raft_consensus.cc:2258] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:03.198551 15639 ts_tablet_manager.cc:1905] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:03.198552 15638 ts_tablet_manager.cc:1905] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:03.201061 15639 ts_tablet_manager.cc:1918] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:53:03.201061 15638 ts_tablet_manager.cc:1918] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 01:53:03.201146 15639 log.cc:1196] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/dfbab53279ef4f54ba7964d61391edab
I20260809 01:53:03.201146 15638 log.cc:1196] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/dfbab53279ef4f54ba7964d61391edab
I20260809 01:53:03.201422 15638 ts_tablet_manager.cc:1939] T dfbab53279ef4f54ba7964d61391edab P a024199de6b147fbac4a3f8292166fe7: Deleting consensus metadata
I20260809 01:53:03.201443 15639 ts_tablet_manager.cc:1939] T dfbab53279ef4f54ba7964d61391edab P c0673839c2c44e13b2b9955b60f3f9d8: Deleting consensus metadata
I20260809 01:53:03.202165 14860 catalog_manager.cc:4841] TS a024199de6b147fbac4a3f8292166fe7 (127.3.220.2:46789): tablet dfbab53279ef4f54ba7964d61391edab (table test-table [id=a2586d75e93e4f70a4d5eaa18a20b0ee]) successfully deleted
I20260809 01:53:03.202373 14860 catalog_manager.cc:4841] TS c0673839c2c44e13b2b9955b60f3f9d8 (127.3.220.3:39969): tablet dfbab53279ef4f54ba7964d61391edab (table test-table [id=a2586d75e93e4f70a4d5eaa18a20b0ee]) successfully deleted
W20260809 01:53:03.206879 14859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:44109 (error 108)
I20260809 01:53:03.207729 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 15196
I20260809 01:53:03.208082 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:03.207590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:44109 (local address 127.0.0.1:38244)
0809 01:53:03.207839 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:03.207936 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:44109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":55,"threads_started":1}
W20260809 01:53:03.211257 14859 catalog_manager.cc:4584] TS 6ea5f5770434470d90b14f06c0884e53 (127.3.220.1:44109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfbab53279ef4f54ba7964d61391edab: Network error: Client connection negotiation failed: client connection to 127.3.220.1:44109: connect: Connection refused (error 111)
I20260809 01:53:03.218987 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 15397
I20260809 01:53:03.232806 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 14836
2026-08-09T01:53:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3501 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:53:03 dist-test-slave-gkw7 krb5kdc[15729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:53:03 dist-test-slave-gkw7 krb5kdc[15729](info): set up 2 sockets
Aug 09 01:53:03 dist-test-slave-gkw7 krb5kdc[15729](info): commencing operation
krb5kdc: starting...
W20260809 01:53:04.471880 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:53:04 dist-test-slave-gkw7 krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:53:04.600445 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:43789
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:43789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:04.714681 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:04.715011 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:04.715085 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:04.719842 15750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:53:04.719971 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:04.720018 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:04.720120 15750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:53:04.720207 15750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:53:04.726272 15750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:43789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:43789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:04.727470 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:04.728629 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:04.737414 15756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:04.737430 15757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:04.737469 15750 server_base.cc:1034] running on GCE node
W20260809 01:53:04.737589 15760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:04.738304 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:04.739379 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:04.740584 15750 hybrid_clock.cc:648] HybridClock initialized: now 1786240384740564 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T01:53:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1515 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:53:04 dist-test-slave-gkw7 krb5kdc[15768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:53:04 dist-test-slave-gkw7 krb5kdc[15768](info): set up 2 sockets
Aug 09 01:53:04 dist-test-slave-gkw7 krb5kdc[15768](info): commencing operation
krb5kdc: starting...
W20260809 01:53:06.050629 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:53:06 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:53:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:53:06.181047 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:45159
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:45159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:06.307402 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:06.307723 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:06.307799 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:06.311965 15784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:53:06.312048 15784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:06.312098 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:06.312166 15784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:53:06.312203 15784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:53:06.317692 15784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:45159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:45159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:06.318782 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:06.319826 15784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:06.328773 15794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:06.328809 15796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:06.328876 15784 server_base.cc:1034] running on GCE node
W20260809 01:53:06.328819 15793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:06.329730 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:06.330889 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:06.332082 15784 hybrid_clock.cc:648] HybridClock initialized: now 1786240386332049 us; error 42 us; skew 500 ppm
Aug 09 01:53:06 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240386, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:06.334957 15784 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:53:06.335830 15784 webserver.cc:460] Webserver started at http://127.3.220.62:43223/ using document root <none> and password file <none>
I20260809 01:53:06.336406 15784 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:06.336503 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:06.336742 15784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:06.338975 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "46e9ec109c57470892d5b0c753172a09"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "04c7cd48ffa5c514790890b0f4fe89fa"
server_key_iv: "0043d1a033595ad6569cf18270b6d21c"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.339560 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "46e9ec109c57470892d5b0c753172a09"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "04c7cd48ffa5c514790890b0f4fe89fa"
server_key_iv: "0043d1a033595ad6569cf18270b6d21c"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.343369 15784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 01:53:06.345950 15805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:06.347203 15784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:53:06.347348 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "46e9ec109c57470892d5b0c753172a09"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "04c7cd48ffa5c514790890b0f4fe89fa"
server_key_iv: "0043d1a033595ad6569cf18270b6d21c"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.347460 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:06.362462 15784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:06.365898 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:06.366112 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:06.373503 15784 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:45159
I20260809 01:53:06.373549 15884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:45159 every 8 connection(s)
I20260809 01:53:06.374550 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:53:06.377358 15885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:06.378541 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 15784
I20260809 01:53:06.378682 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:53:06.379004 3952 external_mini_cluster.cc:1428] Setting key 2eede762d58fef3e5322ba9aded4a3d0
I20260809 01:53:06.384425 15885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Bootstrap starting.
I20260809 01:53:06.387100 15885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:06.387853 15885 log.cc:824] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:53:06 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:53:06.390189 15885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: No bootstrap required, opened a new log
I20260809 01:53:06.394611 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:06.380533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.3.220.62:45159)
0809 01:53:06.380985 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:06.381005 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:06.381050 (+ 45us) server_negotiation.cc:407] Connection header received
0809 01:53:06.381946 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:06.381969 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:06.382404 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:06.382783 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:06.383488 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.384920 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.385843 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.386521 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.386679 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:06.389363 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:06.389370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:06.389386 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:06.389415 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:06.391400 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:06.391834 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:06.391842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:06.391848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:06.391991 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:06.392253 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:06.392258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:06.392261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:06.392866 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:06.393068 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:06.393257 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 01:53:06.393580 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260809 01:53:06.394925 15885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } }
I20260809 01:53:06.395146 15885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:06.395267 15885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9ec109c57470892d5b0c753172a09, State: Initialized, Role: FOLLOWER
I20260809 01:53:06.395872 15885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } }
I20260809 01:53:06.396054 15885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:53:06.396126 15885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:53:06.396219 15885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:06.397102 15885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } }
I20260809 01:53:06.397449 15885 leader_election.cc:302] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46e9ec109c57470892d5b0c753172a09; no voters:
I20260809 01:53:06.397794 15885 leader_election.cc:288] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:53:06.397915 15891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:06.398159 15891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 1 LEADER]: Becoming Leader. State: Replica: 46e9ec109c57470892d5b0c753172a09, State: Running, Role: LEADER
I20260809 01:53:06.398546 15891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } }
I20260809 01:53:06.399026 15885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:53:06.400223 15894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46e9ec109c57470892d5b0c753172a09. Latest consensus state: current_term: 1 leader_uuid: "46e9ec109c57470892d5b0c753172a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } } }
I20260809 01:53:06.400362 15894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:06.400753 15893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46e9ec109c57470892d5b0c753172a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } } }
I20260809 01:53:06.400905 15893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:06.400766 15901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:53:06.404642 15901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:53:06.409845 15901 catalog_manager.cc:1347] Generated new cluster ID: a0de1497e3464f68887dc46e22e90622
I20260809 01:53:06.409917 15901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:53:06.415666 15901 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:53:06.416550 15901 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:53:06.427415 15901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Generated new TSK 0
I20260809 01:53:06.428108 15901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:53:06.464306 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45159
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:06.578208 15918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:06.578498 15918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:06.578574 15918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:06.582561 15918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:06.582643 15918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:06.582733 15918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:53:06.587893 15918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:06.588994 15918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:06.590032 15918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:06.599614 15925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:06.599669 15926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:06.599622 15929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:06.599874 15918 server_base.cc:1034] running on GCE node
I20260809 01:53:06.600309 15918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:06.601011 15918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:06.602200 15918 hybrid_clock.cc:648] HybridClock initialized: now 1786240386602174 us; error 48 us; skew 500 ppm
Aug 09 01:53:06 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240386, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:06.604965 15918 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:53:06.605969 15918 webserver.cc:460] Webserver started at http://127.3.220.1:34457/ using document root <none> and password file <none>
I20260809 01:53:06.606542 15918 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:06.606633 15918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:06.606876 15918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:06.608919 15918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "a34ad37ac505437caca7a4c7a076e2ea"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "c2cf99a469250108a17836e58fca7eb4"
server_key_iv: "c5636bd412b230549282855aa5c60240"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.609493 15918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "a34ad37ac505437caca7a4c7a076e2ea"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "c2cf99a469250108a17836e58fca7eb4"
server_key_iv: "c5636bd412b230549282855aa5c60240"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.613070 15918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:53:06.615520 15943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:06.616714 15918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 01:53:06.616808 15918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "a34ad37ac505437caca7a4c7a076e2ea"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "c2cf99a469250108a17836e58fca7eb4"
server_key_iv: "c5636bd412b230549282855aa5c60240"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.616889 15918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:06.632917 15918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:06.636608 15918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:06.636778 15918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:06.637341 15918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:06.638252 15918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:06.638303 15918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:06.638347 15918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:06.638375 15918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:06.648787 15918 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:40421
I20260809 01:53:06.648820 16093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:40421 every 8 connection(s)
I20260809 01:53:06.649914 15918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:53:06.651202 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 15918
I20260809 01:53:06.651346 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:53:06.651726 3952 external_mini_cluster.cc:1428] Setting key e8e5b38e430f2b228b521ccfa5e0549e
Aug 09 01:53:06 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240386, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:53:06.665906 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:06.651984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:37727 (local address 127.3.220.62:45159)
0809 01:53:06.652184 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:06.652190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:06.653747 (+ 1557us) server_negotiation.cc:407] Connection header received
0809 01:53:06.654730 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:06.654734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:06.654790 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:06.654884 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:06.656660 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.657189 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.658558 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.658857 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.658987 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:06.661823 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:06.661848 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:06.661851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:06.661878 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:06.663509 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:06.664109 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:06.664113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:06.664114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:06.664200 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:06.664592 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:06.664595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:06.664597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:06.664817 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:06.664940 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:06.665517 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 01:53:06.665703 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:53:06.666807 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:06.652469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:37727)
0809 01:53:06.653574 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:06.653637 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:06.654508 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:06.655080 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:06.655110 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:06.655703 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:06.656470 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:06.656499 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.657314 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.657317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:06.658406 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:06.658417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.659034 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.659039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:06.659227 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:06.659908 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:06.659926 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:06.661643 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:06.663653 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:06.663662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:06.663680 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:06.663977 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:06.664305 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:06.664310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:06.664312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:06.664475 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:06.664940 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:06.664949 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:06.665240 (+ 291us) client_negotiation.cc:769] Sending connection context
0809 01:53:06.665452 (+ 212us) client_negotiation.cc:240] Negotiation successful
0809 01:53:06.665865 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":667,"threads_started":1}
I20260809 01:53:06.668238 16094 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45159
I20260809 01:53:06.668514 16094 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:06.669055 16094 heartbeater.cc:511] Master 127.3.220.62:45159 requested a full tablet report, sending...
I20260809 01:53:06.670763 15828 ts_manager.cc:194] Registered new tserver with Master: a34ad37ac505437caca7a4c7a076e2ea (127.3.220.1:40421)
I20260809 01:53:06.672014 15828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:37727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:53:06.682343 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45159
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:06.799019 16102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:06.799268 16102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:06.799346 16102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:06.803514 16102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:06.803575 16102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:06.803696 16102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:53:06.808960 16102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:06.810081 16102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:06.811133 16102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:06.820868 16109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:06.820878 16108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:06.820874 16111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:06.821385 16102 server_base.cc:1034] running on GCE node
I20260809 01:53:06.821807 16102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:06.822448 16102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:06.823608 16102 hybrid_clock.cc:648] HybridClock initialized: now 1786240386823593 us; error 27 us; skew 500 ppm
Aug 09 01:53:06 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240386, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:06.826443 16102 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:53:06.827306 16102 webserver.cc:460] Webserver started at http://127.3.220.2:37093/ using document root <none> and password file <none>
I20260809 01:53:06.827881 16102 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:06.827975 16102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:06.828212 16102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:06.830441 16102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "a6c66468f26d41ecb05e7e353750568d"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "3fc6aa2fd7cc4cab4ebd032f503d7c65"
server_key_iv: "210aebe40aebf176cb5ad716a75c4d18"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.831023 16102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "a6c66468f26d41ecb05e7e353750568d"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "3fc6aa2fd7cc4cab4ebd032f503d7c65"
server_key_iv: "210aebe40aebf176cb5ad716a75c4d18"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.834789 16102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:53:06.837231 16122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:06.838652 16102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:53:06.838792 16102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "a6c66468f26d41ecb05e7e353750568d"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "3fc6aa2fd7cc4cab4ebd032f503d7c65"
server_key_iv: "210aebe40aebf176cb5ad716a75c4d18"
server_key_version: "encryptionkey@0"
I20260809 01:53:06.838907 16102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:06.874198 16102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:06.877766 16102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:06.878021 16102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:06.878672 16102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:06.879546 16102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:06.879617 16102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:06.879691 16102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:06.879745 16102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:06.888550 16102 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:46749
I20260809 01:53:06.888574 16291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:46749 every 8 connection(s)
I20260809 01:53:06.889739 16102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:53:06.890020 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 16102
I20260809 01:53:06.890164 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:53:06.890519 3952 external_mini_cluster.cc:1428] Setting key 15ec8005fde66681649729057a17564f
Aug 09 01:53:06 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240386, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:53:06.906355 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:06.891723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:55323 (local address 127.3.220.62:45159)
0809 01:53:06.891867 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:06.891871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:06.892897 (+ 1026us) server_negotiation.cc:407] Connection header received
0809 01:53:06.894385 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:06.894392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:06.894458 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:06.894521 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:06.896432 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.896956 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.898881 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.899137 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.899237 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:06.902597 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:06.902601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:06.902603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:06.902626 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:06.904079 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:06.904639 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:06.904642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:06.904643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:06.904717 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:06.905114 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:06.905117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:06.905119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:06.905383 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:06.905495 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:06.906126 (+ 631us) server_negotiation.cc:299] Negotiation successful
0809 01:53:06.906226 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:53:06.907570 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:06.892261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:55323)
0809 01:53:06.892754 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:06.892799 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:06.894110 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:06.894967 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:06.894977 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:06.895534 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:06.896246 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:06.896269 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.897099 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.897106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:06.898753 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:06.898767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:06.899325 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:06.899332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:06.899576 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:06.900308 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:06.900329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:06.902396 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:06.904188 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:06.904193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:06.904205 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:06.904529 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:06.904792 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:06.904795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:06.904797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:06.904992 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:06.905494 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:06.905499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:06.905851 (+ 352us) client_negotiation.cc:769] Sending connection context
0809 01:53:06.906063 (+ 212us) client_negotiation.cc:240] Negotiation successful
0809 01:53:06.906440 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":152,"threads_started":1}
I20260809 01:53:06.908842 16292 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45159
I20260809 01:53:06.909094 16292 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:06.909657 16292 heartbeater.cc:511] Master 127.3.220.62:45159 requested a full tablet report, sending...
I20260809 01:53:06.910796 15828 ts_manager.cc:194] Registered new tserver with Master: a6c66468f26d41ecb05e7e353750568d (127.3.220.2:46749)
I20260809 01:53:06.911379 15828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:55323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:53:06.921059 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45159
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:07.035688 16299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:07.035967 16299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:07.036011 16299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:07.040318 16299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:07.040383 16299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:07.040477 16299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:53:07.046118 16299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:07.047313 16299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:07.048383 16299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:07.058224 16308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.058236 16306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:07.058588 16299 server_base.cc:1034] running on GCE node
W20260809 01:53:07.058236 16313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:07.059043 16299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:07.059679 16299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:07.060873 16299 hybrid_clock.cc:648] HybridClock initialized: now 1786240387060857 us; error 27 us; skew 500 ppm
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240387, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:07.063802 16299 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:53:07.064677 16299 webserver.cc:460] Webserver started at http://127.3.220.3:35131/ using document root <none> and password file <none>
I20260809 01:53:07.065244 16299 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:07.065352 16299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:07.065646 16299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:07.067703 16299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "727bd8c440e24001aa9e603227be0b66"
format_stamp: "Formatted at 2026-08-09 01:53:07 on dist-test-slave-gkw7"
server_key: "74ea563ab1e4eb52c9cd88199ac7785c"
server_key_iv: "ade61a6f47eca41352d9f12335e29b19"
server_key_version: "encryptionkey@0"
I20260809 01:53:07.068289 16299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "727bd8c440e24001aa9e603227be0b66"
format_stamp: "Formatted at 2026-08-09 01:53:07 on dist-test-slave-gkw7"
server_key: "74ea563ab1e4eb52c9cd88199ac7785c"
server_key_iv: "ade61a6f47eca41352d9f12335e29b19"
server_key_version: "encryptionkey@0"
I20260809 01:53:07.072105 16299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:53:07.074637 16323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.075865 16299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:53:07.076010 16299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "727bd8c440e24001aa9e603227be0b66"
format_stamp: "Formatted at 2026-08-09 01:53:07 on dist-test-slave-gkw7"
server_key: "74ea563ab1e4eb52c9cd88199ac7785c"
server_key_iv: "ade61a6f47eca41352d9f12335e29b19"
server_key_version: "encryptionkey@0"
I20260809 01:53:07.076124 16299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:07.097769 16299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:07.101372 16299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:07.101608 16299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:07.102206 16299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:07.103026 16299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:07.103075 16299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.103116 16299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:07.103134 16299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.111753 16299 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:40145
I20260809 01:53:07.111778 16478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:40145 every 8 connection(s)
I20260809 01:53:07.112658 16299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:53:07.118561 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 16299
I20260809 01:53:07.118701 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:53:07.119026 3952 external_mini_cluster.cc:1428] Setting key 5ec07c109bcec178e3e7a233b0ed5276
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240387, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:53:07.127211 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:07.114563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:33379 (local address 127.3.220.62:45159)
0809 01:53:07.114712 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:07.114715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:07.115619 (+ 904us) server_negotiation.cc:407] Connection header received
0809 01:53:07.116955 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:07.116959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:07.117020 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:07.117128 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:07.118420 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.119094 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.120760 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.121062 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.121171 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.123733 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:07.123736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:07.123738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:07.123763 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:07.125026 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:07.125510 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:07.125512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:07.125514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:07.125650 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:07.126060 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:07.126064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:07.126066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:07.126264 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:07.126382 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:07.126899 (+ 517us) server_negotiation.cc:299] Negotiation successful
0809 01:53:07.127037 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 01:53:07.128057 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:07.114967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:33379)
0809 01:53:07.115468 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:07.115510 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:07.116727 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:07.117315 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:07.117323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:07.117765 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:07.118260 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.118273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.119236 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.119239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.120638 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.120649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.121249 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.121254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.121456 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.122085 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:07.122101 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:07.123566 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:07.125133 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:07.125138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:07.125150 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:07.125423 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:07.125752 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:07.125755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:07.125757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:07.125955 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:07.126380 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:07.126388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:07.126653 (+ 265us) client_negotiation.cc:769] Sending connection context
0809 01:53:07.126854 (+ 201us) client_negotiation.cc:240] Negotiation successful
0809 01:53:07.127191 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20260809 01:53:07.129179 16481 heartbeater.cc:347] Connected to a master server at 127.3.220.62:45159
I20260809 01:53:07.129443 16481 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:07.129959 16481 heartbeater.cc:511] Master 127.3.220.62:45159 requested a full tablet report, sending...
I20260809 01:53:07.130997 15828 ts_manager.cc:194] Registered new tserver with Master: 727bd8c440e24001aa9e603227be0b66 (127.3.220.3:40145)
I20260809 01:53:07.131575 15828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:33379
I20260809 01:53:07.133033 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab.
Entry for principal oryx/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab.
WARNING: no policy specified for oryx/127.3.220.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab.
Entry for principal oryx/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab.
WARNING: no policy specified for oryx/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab.
Entry for principal oryx/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab.
I20260809 01:53:07.214871 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 15918
I20260809 01:53:07.226374 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 16102
I20260809 01:53:07.239387 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 16299
I20260809 01:53:07.256392 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 15784
I20260809 01:53:07.267802 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:45159
--webserver_interface=127.3.220.62
--webserver_port=43223
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:45159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:07.385219 16505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:07.385494 16505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:07.385602 16505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:07.389995 16505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:53:07.390074 16505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:07.390136 16505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:07.390180 16505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:53:07.390224 16505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:53:07.395839 16505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:45159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:45159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:07.396978 16505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:07.398154 16505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:07.407130 16510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.407145 16516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.407145 16511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:07.407410 16505 server_base.cc:1034] running on GCE node
I20260809 01:53:07.407985 16505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:07.409065 16505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:07.410254 16505 hybrid_clock.cc:648] HybridClock initialized: now 1786240387410241 us; error 25 us; skew 500 ppm
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240387, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:07.413080 16505 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:53:07.413980 16505 webserver.cc:460] Webserver started at http://127.3.220.62:43223/ using document root <none> and password file <none>
I20260809 01:53:07.414563 16505 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:07.414638 16505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:07.418437 16505 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:53:07.420462 16527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.421772 16505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:53:07.421933 16505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "46e9ec109c57470892d5b0c753172a09"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "04c7cd48ffa5c514790890b0f4fe89fa"
server_key_iv: "0043d1a033595ad6569cf18270b6d21c"
server_key_version: "encryptionkey@0"
I20260809 01:53:07.422461 16505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:07.431577 16505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:07.435005 16505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:07.435166 16505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:07.442495 16505 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:45159
I20260809 01:53:07.442571 16598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:45159 every 8 connection(s)
I20260809 01:53:07.443614 16505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:53:07.444907 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 16505
I20260809 01:53:07.445354 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:40421
--local_ip_for_outbound_sockets=127.3.220.1
--tserver_master_addrs=127.3.220.62:45159
--webserver_port=34457
--webserver_interface=127.3.220.1
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.220.1
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:53:07.448673 16600 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:53:07.451296 16600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Bootstrap starting.
I20260809 01:53:07.474937 16600 log.cc:824] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:07.483248 16600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:53:07.483701 16600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Bootstrap complete.
I20260809 01:53:07.489476 16600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } }
I20260809 01:53:07.490038 16600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9ec109c57470892d5b0c753172a09, State: Initialized, Role: FOLLOWER
I20260809 01:53:07.490626 16600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } }
I20260809 01:53:07.490813 16600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:53:07.490900 16600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:53:07.491021 16600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:53:07.493661 16600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } }
I20260809 01:53:07.494052 16600 leader_election.cc:302] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46e9ec109c57470892d5b0c753172a09; no voters:
I20260809 01:53:07.494414 16600 leader_election.cc:288] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 01:53:07.494552 16605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 01:53:07.494830 16605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [term 2 LEADER]: Becoming Leader. State: Replica: 46e9ec109c57470892d5b0c753172a09, State: Running, Role: LEADER
I20260809 01:53:07.495164 16605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } }
I20260809 01:53:07.495754 16600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:53:07.496050 16607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46e9ec109c57470892d5b0c753172a09. Latest consensus state: current_term: 2 leader_uuid: "46e9ec109c57470892d5b0c753172a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } } }
I20260809 01:53:07.496177 16607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:07.496526 16606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "46e9ec109c57470892d5b0c753172a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e9ec109c57470892d5b0c753172a09" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 45159 } } }
I20260809 01:53:07.496613 16606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:07.497164 16611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:53:07.500798 16611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:53:07.501791 16611 catalog_manager.cc:1259] Loaded cluster ID: a0de1497e3464f68887dc46e22e90622
I20260809 01:53:07.501888 16611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:53:07.503538 16611 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:53:07.504106 16611 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46e9ec109c57470892d5b0c753172a09: Loaded TSK: 0
I20260809 01:53:07.505045 16611 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 01:53:07.588431 16602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:07.588694 16602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:07.588737 16602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:07.592743 16602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:07.592803 16602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:07.592898 16602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:53:07.598378 16602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.1.keytab
--principal=oryx/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:40421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=34457
--tserver_master_addrs=127.3.220.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:07.599462 16602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:07.600458 16602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:07.609912 16633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.609972 16634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.610018 16636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:07.610443 16602 server_base.cc:1034] running on GCE node
I20260809 01:53:07.610862 16602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:07.611515 16602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:07.612697 16602 hybrid_clock.cc:648] HybridClock initialized: now 1786240387612675 us; error 37 us; skew 500 ppm
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240387, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:07.615648 16602 init.cc:377] Logged in from keytab as oryx/127.3.220.1@KRBTEST.COM (short username oryx)
I20260809 01:53:07.616562 16602 webserver.cc:460] Webserver started at http://127.3.220.1:34457/ using document root <none> and password file <none>
I20260809 01:53:07.617188 16602 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:07.617306 16602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:07.621490 16602 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:53:07.623756 16645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.625037 16602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:53:07.625172 16602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "a34ad37ac505437caca7a4c7a076e2ea"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "c2cf99a469250108a17836e58fca7eb4"
server_key_iv: "c5636bd412b230549282855aa5c60240"
server_key_version: "encryptionkey@0"
I20260809 01:53:07.625862 16602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:07.637631 16602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:07.640748 16602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:07.640956 16602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:07.641626 16602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:07.643030 16602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:07.643117 16602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.643213 16602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:07.643242 16602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.654879 16602 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:40421
I20260809 01:53:07.654909 16798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:40421 every 8 connection(s)
I20260809 01:53:07.655951 16602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:53:07.663763 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 16602
I20260809 01:53:07.664302 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:46749
--local_ip_for_outbound_sockets=127.3.220.2
--tserver_master_addrs=127.3.220.62:45159
--webserver_port=37093
--webserver_interface=127.3.220.2
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.220.2
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:07.671655 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:07.658540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:53069)
0809 01:53:07.659159 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:07.659217 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:07.660555 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:07.661851 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:07.661858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:07.662274 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:07.662780 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.662794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.664305 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.664309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.665887 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.665904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.666941 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.666945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.667087 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.667748 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:07.667765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:07.670667 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":149,"threads_started":1}
W20260809 01:53:07.671648 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:07.658426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:53069 (local address 127.3.220.62:45159)
0809 01:53:07.658944 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:07.658962 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:07.659348 (+ 386us) server_negotiation.cc:407] Connection header received
0809 01:53:07.661002 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:07.661018 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:07.661384 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:07.661759 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:07.662962 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.664148 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.666023 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.666769 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.666946 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.670890 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:53069: BlockingRecv error: recv got EOF from 127.3.220.1:53069 (error 108)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":155,"threads_started":1}
W20260809 01:53:07.672250 16799 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:45159 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.220.62:45159: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:07.677683 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:07.672687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:52397)
0809 01:53:07.672817 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:07.672838 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:07.672929 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:07.673186 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:07.673189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:07.673340 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:07.673423 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.673427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.674138 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.674144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.674690 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.674697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.675136 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.675140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.675238 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.675730 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:07.675741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:07.677521 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 01:53:07.677788 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:07.672833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:52397 (local address 127.3.220.62:45159)
0809 01:53:07.672973 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:07.672979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:07.672993 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:53:07.673037 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:07.673039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:07.673075 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:07.673170 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:07.673562 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.674036 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.674808 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.675027 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.675134 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.677657 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:52397: BlockingRecv error: recv got EOF from 127.3.220.1:52397 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:07.683197 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:07.678184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:52981)
0809 01:53:07.678406 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:07.678418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:07.678521 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:07.678760 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:07.678763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:07.678915 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:07.678977 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.678981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.679751 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.679756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.680269 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.680277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.680707 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.680710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.680783 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.681251 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:07.681263 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:07.683068 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161}
W20260809 01:53:07.683310 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:07.678292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:52981 (local address 127.3.220.62:45159)
0809 01:53:07.678387 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:07.678391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:07.678486 (+ 95us) server_negotiation.cc:407] Connection header received
0809 01:53:07.678617 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:07.678620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:07.678659 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:07.678746 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:07.679085 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.679652 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.680388 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.680598 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.680724 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.683177 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:52981: BlockingRecv error: recv got EOF from 127.3.220.1:52981 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260809 01:53:07.683475 16799 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 01:53:07.780887 16804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:07.781145 16804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:07.781188 16804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:07.785243 16804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:07.785322 16804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:07.785431 16804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:53:07.790736 16804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.2.keytab
--principal=oryx/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:46749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=37093
--tserver_master_addrs=127.3.220.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:07.791811 16804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:07.792857 16804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:07.802902 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.802891 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.802894 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:07.803293 16804 server_base.cc:1034] running on GCE node
I20260809 01:53:07.803745 16804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:07.804337 16804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:07.805498 16804 hybrid_clock.cc:648] HybridClock initialized: now 1786240387805483 us; error 25 us; skew 500 ppm
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240387, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:07.808377 16804 init.cc:377] Logged in from keytab as oryx/127.3.220.2@KRBTEST.COM (short username oryx)
I20260809 01:53:07.809239 16804 webserver.cc:460] Webserver started at http://127.3.220.2:37093/ using document root <none> and password file <none>
I20260809 01:53:07.809880 16804 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:07.809985 16804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:07.813930 16804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 01:53:07.815936 16824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.817198 16804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:53:07.817338 16804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "a6c66468f26d41ecb05e7e353750568d"
format_stamp: "Formatted at 2026-08-09 01:53:06 on dist-test-slave-gkw7"
server_key: "3fc6aa2fd7cc4cab4ebd032f503d7c65"
server_key_iv: "210aebe40aebf176cb5ad716a75c4d18"
server_key_version: "encryptionkey@0"
I20260809 01:53:07.817945 16804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:07.831290 16804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:07.834602 16804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:07.834841 16804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:07.835450 16804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:07.836304 16804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:07.836391 16804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.836474 16804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:07.836526 16804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:07.845715 16804 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:46749
I20260809 01:53:07.845786 16984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:46749 every 8 connection(s)
I20260809 01:53:07.846946 16804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:53:07.852653 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 16804
I20260809 01:53:07.853127 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:40145
--local_ip_for_outbound_sockets=127.3.220.3
--tserver_master_addrs=127.3.220.62:45159
--webserver_port=35131
--webserver_interface=127.3.220.3
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.220.3
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 01:53:07.859766 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:07.848847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:55131 (local address 127.3.220.62:45159)
0809 01:53:07.849036 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:07.849041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:07.849963 (+ 922us) server_negotiation.cc:407] Connection header received
0809 01:53:07.851238 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:07.851241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:07.851304 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:07.851384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:07.852810 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.853465 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.855289 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.855673 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.855799 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.859572 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:55131: BlockingRecv error: recv got EOF from 127.3.220.2:55131 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 01:53:07.860360 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:07.849273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:55131)
0809 01:53:07.849789 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:07.849834 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:07.851024 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:07.851573 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:07.851580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:07.851983 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:07.852620 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.852645 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.853638 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.853644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.855165 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.855176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.855849 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.855851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.855993 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.856630 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:07.856647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:07.859343 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":117,"threads_started":1}
W20260809 01:53:07.860985 16986 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:45159 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.220.62:45159: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:07.868288 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:07.861479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:37725)
0809 01:53:07.861753 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:07.861769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:07.861883 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:07.862628 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:07.862631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:07.862835 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:07.862956 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.862963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.864087 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.864091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.865081 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.865091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.865748 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.865751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.865823 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.866338 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:07.866355 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:07.868182 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185}
W20260809 01:53:07.868448 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:07.862186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:37725 (local address 127.3.220.62:45159)
0809 01:53:07.862354 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:07.862358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:07.862371 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:53:07.862435 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:07.862438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:07.862480 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:07.862581 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:07.863092 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.863954 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.865203 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.865633 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.865764 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.868336 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:37725: BlockingRecv error: recv got EOF from 127.3.220.2:37725 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:07.874074 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:07.869026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:40921)
0809 01:53:07.869141 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:07.869156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:07.869230 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:07.869464 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:07.869466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:07.869636 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:07.869703 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.869707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.870682 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.870684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.871193 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:07.871200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.871725 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.871727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:07.871785 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.872241 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:07.872253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:07.873989 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 01:53:07.874235 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:07.869113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:40921 (local address 127.3.220.62:45159)
0809 01:53:07.869218 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:07.869222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:07.869238 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:53:07.869313 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:07.869316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:07.869361 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:07.869431 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:07.869814 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.870577 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.871292 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:07.871625 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:07.871719 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:07.874129 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:40921: BlockingRecv error: recv got EOF from 127.3.220.2:40921 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260809 01:53:07.874449 16986 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 01:53:07.971835 16990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:07.972126 16990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:07.972203 16990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:07.976329 16990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:07.976418 16990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:07.976506 16990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:53:07.981814 16990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/krb5kdc/oryx_127.3.220.3.keytab
--principal=oryx/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:40145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=35131
--tserver_master_addrs=127.3.220.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:07.982904 16990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:07.983952 16990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:07.993569 16998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.993621 17004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:07.993628 17001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:07.993896 16990 server_base.cc:1034] running on GCE node
I20260809 01:53:07.994270 16990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:07.994899 16990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:07.996079 16990 hybrid_clock.cc:648] HybridClock initialized: now 1786240387996060 us; error 25 us; skew 500 ppm
Aug 09 01:53:07 dist-test-slave-gkw7 krb5kdc[15768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240387, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:07.999226 16990 init.cc:377] Logged in from keytab as oryx/127.3.220.3@KRBTEST.COM (short username oryx)
I20260809 01:53:08.000134 16990 webserver.cc:460] Webserver started at http://127.3.220.3:35131/ using document root <none> and password file <none>
I20260809 01:53:08.000715 16990 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:08.000794 16990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:08.004895 16990 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:53:08.007135 17013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:08.008411 16990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:53:08.008553 16990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "727bd8c440e24001aa9e603227be0b66"
format_stamp: "Formatted at 2026-08-09 01:53:07 on dist-test-slave-gkw7"
server_key: "74ea563ab1e4eb52c9cd88199ac7785c"
server_key_iv: "ade61a6f47eca41352d9f12335e29b19"
server_key_version: "encryptionkey@0"
I20260809 01:53:08.009099 16990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:08.038069 16990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:08.041276 16990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:08.041491 16990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:08.042174 16990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:08.043061 16990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:08.043135 16990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:08.043211 16990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:08.043253 16990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:08.052986 16990 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:40145
I20260809 01:53:08.053011 17175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:40145 every 8 connection(s)
I20260809 01:53:08.054162 16990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:53:08.061522 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 16990
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 01:53:08.066215 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:08.056026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:46145 (local address 127.3.220.62:45159)
0809 01:53:08.056167 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:08.056171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:08.057013 (+ 842us) server_negotiation.cc:407] Connection header received
0809 01:53:08.058461 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:08.058465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:08.058529 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:08.058629 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:08.059838 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.060394 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.061852 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.062187 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.062326 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.066059 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:46145: BlockingRecv error: recv got EOF from 127.3.220.3:46145 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":53}
I20260809 01:53:08.066923 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:08.056377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:46145)
0809 01:53:08.056849 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:08.056893 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:08.058216 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:08.058827 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:08.058834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:08.059196 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:08.059675 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.059687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.060524 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.060527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.061687 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.061699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.062363 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.062367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.062558 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.063213 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:08.063231 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:08.065916 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
W20260809 01:53:08.067569 17176 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:45159 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.220.62:45159: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
I20260809 01:53:08.072163 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:08.062737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.3.220.62:45159)
0809 01:53:08.063042 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:08.063045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:08.063055 (+ 10us) server_negotiation.cc:407] Connection header received
0809 01:53:08.063084 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:08.063086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:08.063120 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:08.063227 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:08.064040 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.064610 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.065655 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.066024 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.066126 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.067843 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:08.067850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:08.067870 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:08.067902 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:08.069821 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:08.070251 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:08.070257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:08.070264 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:08.070410 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:08.070706 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:08.070709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:08.070710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:08.071144 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:08.071357 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:08.071689 (+ 332us) server_negotiation.cc:299] Negotiation successful
0809 01:53:08.071929 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":241,"thread_start_us":73,"threads_started":1}
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:08.073679 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:08.068052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:46493)
0809 01:53:08.068240 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:08.068257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:08.068323 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:08.068666 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:08.068671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:08.068852 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:08.068985 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.068997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.069766 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.069770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.070540 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.070549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.070988 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.070992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.071085 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.071570 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:08.071584 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:08.073565 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111}
W20260809 01:53:08.074035 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:08.068131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:46493 (local address 127.3.220.62:45159)
0809 01:53:08.068473 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:08.068476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:08.068488 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:53:08.068518 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:08.068520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:08.068553 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:08.068616 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:08.069093 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.069649 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.070648 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.070877 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.070961 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.073698 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:46493: BlockingRecv error: recv got EOF from 127.3.220.3:46493 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":47}
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:08.079107 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:08.074199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:42199)
0809 01:53:08.074323 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:08.074341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:08.074420 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:08.074988 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:08.074991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:08.075124 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:08.075196 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.075201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.075942 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.075944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.076466 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.076474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.076857 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.076860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.076917 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.077324 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:08.077334 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:08.079017 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 01:53:08.079277 16802 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:08.074404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:42199 (local address 127.3.220.62:45159)
0809 01:53:08.074777 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:08.074783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:08.074798 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:53:08.074832 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:08.074835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:08.074872 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:08.074940 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:08.075307 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.075845 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.076565 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.076765 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.076837 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.079159 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:42199: BlockingRecv error: recv got EOF from 127.3.220.3:42199 (error 108)
Metrics: {"server-negotiator.queue_time_us":296}
W20260809 01:53:08.079389 17176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:08.690014 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:08.684172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:34159)
0809 01:53:08.684493 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:08.684507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:08.684607 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:08.684890 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:08.684892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:08.685080 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:08.685179 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.685184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.685959 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.685962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.686538 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.686546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.686995 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.686998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.687088 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.687758 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:08.687799 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:08.689888 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":114,"threads_started":1}
W20260809 01:53:08.690208 17183 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:08.684260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:34159 (local address 127.3.220.62:45159)
0809 01:53:08.684531 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:08.684535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:08.684553 (+ 18us) server_negotiation.cc:407] Connection header received
0809 01:53:08.684705 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:08.684708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:08.684754 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:08.684850 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:08.685290 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.685867 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.686640 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.686867 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.686984 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.690029 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:34159: BlockingRecv error: recv got EOF from 127.3.220.1:34159 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:08 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:08.880401 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:08.875102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:55341)
0809 01:53:08.875390 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:08.875408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:08.875506 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:08.875723 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:08.875725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:08.875909 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:08.876005 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.876010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.876709 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.876711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.877280 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:08.877288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.877794 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.877798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:08.877865 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.878364 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:08.878375 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:08.880267 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
W20260809 01:53:08.880537 17183 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:08.875154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:55341 (local address 127.3.220.62:45159)
0809 01:53:08.875286 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:08.875291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:08.875473 (+ 182us) server_negotiation.cc:407] Connection header received
0809 01:53:08.875572 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:08.875574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:08.875619 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:08.875686 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:08.876099 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.876615 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.877385 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:08.877621 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:08.877696 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:08.880391 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:55341: BlockingRecv error: recv got EOF from 127.3.220.2:55341 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 01:53:09 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:09 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:09.085704 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:09.080055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:33373)
0809 01:53:09.080350 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:09.080367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:09.080491 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:09.080729 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:09.080732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:09.080918 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:09.081043 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:09.081049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.081790 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.081803 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:09.082406 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:09.082413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.082827 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.082830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:09.082892 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:09.083374 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:09.083386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:09.085497 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":72,"threads_started":1}
W20260809 01:53:09.085774 17183 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:09.080182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:33373 (local address 127.3.220.62:45159)
0809 01:53:09.080371 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:09.080375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:09.080463 (+ 88us) server_negotiation.cc:407] Connection header received
0809 01:53:09.080574 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:09.080576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:09.080623 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:09.080688 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:09.081143 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.081697 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.082508 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.082722 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.082800 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:09.085682 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:33373: BlockingRecv error: recv got EOF from 127.3.220.3:33373 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 09 01:53:09 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:09 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:09.696869 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:09.690895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:57839)
0809 01:53:09.691191 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:09.691205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:09.691301 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:09.691548 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:09.691550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:09.691732 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:09.691836 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:09.691841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.692615 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.692619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:09.693320 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:09.693331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.693983 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.693987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:09.694079 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:09.694586 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:09.694598 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:09.696646 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
W20260809 01:53:09.696907 17189 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:09.690996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:57839 (local address 127.3.220.62:45159)
0809 01:53:09.691272 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:09.691276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:09.691307 (+ 31us) server_negotiation.cc:407] Connection header received
0809 01:53:09.691386 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:09.691388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:09.691432 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:09.691510 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:09.691937 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.692483 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.693491 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.693840 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.693948 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:09.696780 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:57839: BlockingRecv error: recv got EOF from 127.3.220.1:57839 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
Aug 09 01:53:09 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:09 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:09.891076 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:09.881275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:38209)
0809 01:53:09.881644 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:09.881657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:09.881789 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:09.882076 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:09.882079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:09.882292 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:09.882387 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:09.882393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.883149 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.883153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:09.883708 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:09.883716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.884199 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.884202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:09.884287 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:09.888710 (+ 4423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:09.888723 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:09.890867 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":112,"threads_started":1}
W20260809 01:53:09.891163 17189 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:09.881372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:38209 (local address 127.3.220.62:45159)
0809 01:53:09.881569 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:09.881574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:09.881727 (+ 153us) server_negotiation.cc:407] Connection header received
0809 01:53:09.881903 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:09.881907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:09.881960 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:09.882052 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:09.882500 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.883034 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.883844 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:09.884075 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:09.884184 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:09.891043 (+ 6859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:38209: BlockingRecv error: recv got EOF from 127.3.220.2:38209 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 01:53:10 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:10 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:10.092173 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:10.086488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:36663)
0809 01:53:10.086809 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:10.086827 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:10.086979 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:10.087271 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:10.087289 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:10.087497 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:10.087591 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:10.087598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.088389 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.088393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:10.088962 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:10.088970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.089452 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.089456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:10.089595 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:10.090161 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:10.090173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:10.091970 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":89,"threads_started":1}
W20260809 01:53:10.092212 17189 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:10.086676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:36663 (local address 127.3.220.62:45159)
0809 01:53:10.086796 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:10.086801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:10.086918 (+ 117us) server_negotiation.cc:407] Connection header received
0809 01:53:10.087093 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:10.087096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:10.087150 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:10.087244 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:10.087719 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.088261 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.089087 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.089334 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.089448 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:10.092099 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:36663: BlockingRecv error: recv got EOF from 127.3.220.3:36663 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 01:53:10 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:10 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:10.703787 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:10.697889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:38905)
0809 01:53:10.698229 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:10.698246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:10.698394 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:10.698669 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:10.698672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:10.698889 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:10.698985 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:10.698990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.699779 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.699793 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:10.700364 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:10.700372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.700899 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.700902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:10.700987 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:10.701617 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:10.701631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:10.703642 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":133,"threads_started":1}
W20260809 01:53:10.703967 17194 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:10.697935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:38905 (local address 127.3.220.62:45159)
0809 01:53:10.698228 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:10.698234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:10.698329 (+ 95us) server_negotiation.cc:407] Connection header received
0809 01:53:10.698487 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:10.698491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:10.698540 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:10.698648 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:10.699106 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.699664 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.700504 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.700759 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.700885 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:10.703791 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:38905: BlockingRecv error: recv got EOF from 127.3.220.1:38905 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
Aug 09 01:53:10 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:10 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:10.898540 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:10.892046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:43079)
0809 01:53:10.892454 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:10.892469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:10.892585 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:10.893051 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:10.893054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:10.893339 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:10.893464 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:10.893476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.894498 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.894504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:10.895078 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:10.895086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.895577 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.895581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:10.895666 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:10.896252 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:10.896265 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:10.898342 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":86,"threads_started":1}
W20260809 01:53:10.898577 17194 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:10.892121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:43079 (local address 127.3.220.62:45159)
0809 01:53:10.892292 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:10.892296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:10.892553 (+ 257us) server_negotiation.cc:407] Connection header received
0809 01:53:10.892703 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:10.892708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:10.892762 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:10.892873 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:10.893738 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.894381 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.895235 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:10.895460 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:10.895572 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:10.898471 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:43079: BlockingRecv error: recv got EOF from 127.3.220.2:43079 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 09 01:53:11 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:11 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:11.099469 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:11.093097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:58037)
0809 01:53:11.093485 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:11.093498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:11.093660 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:11.094002 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:11.094005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:11.094344 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:11.094462 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:11.094468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.095207 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.095210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:11.095907 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:11.095917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.096375 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.096379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:11.096527 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:11.097225 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:11.097239 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:11.099320 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
W20260809 01:53:11.099613 17194 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:11.093208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:58037 (local address 127.3.220.62:45159)
0809 01:53:11.093376 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:11.093381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:11.093594 (+ 213us) server_negotiation.cc:407] Connection header received
0809 01:53:11.093770 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:11.093773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:11.093851 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:11.093931 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:11.094575 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.095105 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.096012 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.096231 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.096320 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:11.099460 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:58037: BlockingRecv error: recv got EOF from 127.3.220.3:58037 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 01:53:11 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:11 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:11.710350 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:11.704572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:54483)
0809 01:53:11.704866 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:11.704883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:11.704987 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:11.705273 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:11.705278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:11.705485 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:11.705658 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:11.705665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.706450 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.706453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:11.707024 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:11.707032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.707500 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.707503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:11.707573 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:11.708190 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:11.708202 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:11.710214 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
W20260809 01:53:11.710523 17198 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:11.704734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:54483 (local address 127.3.220.62:45159)
0809 01:53:11.704989 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:11.704993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:11.705006 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:53:11.705046 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:11.705048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:11.705089 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:11.705169 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:11.705779 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.706333 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.707142 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.707374 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.707454 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:11.710376 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:54483: BlockingRecv error: recv got EOF from 127.3.220.1:54483 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
Aug 09 01:53:11 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:11 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:11.905683 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:11.899503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:55385)
0809 01:53:11.899810 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:11.899824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:11.899935 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:11.900209 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:11.900212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:11.900415 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:11.900520 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:11.900525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.901316 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.901319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:11.901979 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:11.901988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.902444 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.902447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:11.902513 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:11.903089 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:11.903102 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:11.905451 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
W20260809 01:53:11.905903 17198 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:11.899624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:55385 (local address 127.3.220.62:45159)
0809 01:53:11.899784 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:11.899788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:11.899902 (+ 114us) server_negotiation.cc:407] Connection header received
0809 01:53:11.900021 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:11.900023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:11.900071 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:11.900171 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:11.900665 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.901206 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.902092 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:11.902336 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:11.902420 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:11.905735 (+ 3315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:55385: BlockingRecv error: recv got EOF from 127.3.220.2:55385 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 01:53:12 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:12 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:12.107278 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:12.100382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:36153)
0809 01:53:12.100660 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:12.100674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:12.100807 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:12.101128 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:12.101130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:12.101300 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:12.101396 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:12.101401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.102685 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.102688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:12.103300 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:12.103309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.104014 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.104017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:12.104083 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:12.104676 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:12.104689 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:12.107124 (+ 2435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
W20260809 01:53:12.107471 17198 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:12.100473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:36153 (local address 127.3.220.62:45159)
0809 01:53:12.100635 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:12.100641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:12.100774 (+ 133us) server_negotiation.cc:407] Connection header received
0809 01:53:12.100923 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:12.100928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:12.100998 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:12.101098 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:12.101549 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.102557 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.103433 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.103888 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.104020 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:12.107304 (+ 3284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:36153: BlockingRecv error: recv got EOF from 127.3.220.3:36153 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 01:53:12 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:12 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:12.718842 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:12.711420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:48855)
0809 01:53:12.711782 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:12.711801 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:12.711922 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:12.712371 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:12.712376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:12.712624 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:12.712740 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:12.712747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.714052 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.714056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:12.714668 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:12.714678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.715372 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.715376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:12.715462 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:12.716039 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:12.716054 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:12.718642 (+ 2588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":115,"threads_started":1}
W20260809 01:53:12.718986 17204 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:12.711810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:48855 (local address 127.3.220.62:45159)
0809 01:53:12.712086 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:12.712091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:12.712109 (+ 18us) server_negotiation.cc:407] Connection header received
0809 01:53:12.712163 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:12.712166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:12.712218 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:12.712331 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:12.712879 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.713804 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.714808 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.715154 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.715271 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:12.718828 (+ 3557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:48855: BlockingRecv error: recv got EOF from 127.3.220.1:48855 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
Aug 09 01:53:12 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:12 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:12.912236 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:12.906480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:58505)
0809 01:53:12.906756 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:12.906770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:12.906853 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:12.907082 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:12.907084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:12.907272 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:12.907378 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:12.907384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.908184 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.908187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:12.908767 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:12.908775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.909239 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.909241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:12.909311 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:12.909899 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:12.909911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:12.912008 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":104,"threads_started":1}
W20260809 01:53:12.912355 17204 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:12.906684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:58505 (local address 127.3.220.62:45159)
0809 01:53:12.906841 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:12.906845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:12.906859 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:53:12.906923 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:12.906925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:12.906972 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:12.907049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:12.907496 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.908064 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.908890 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:12.909130 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:12.909225 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:12.912235 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:58505: BlockingRecv error: recv got EOF from 127.3.220.2:58505 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 01:53:13 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:13 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:13.114158 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:13.108411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:44001)
0809 01:53:13.108725 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:13.108739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:13.108838 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:13.109103 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:13.109107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:13.109308 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:13.109407 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:13.109413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.110201 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.110204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:13.110817 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:13.110827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.111278 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.111282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:13.111361 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:13.111849 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:13.111861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:13.113997 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
W20260809 01:53:13.114301 17204 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:13.108555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:44001 (local address 127.3.220.62:45159)
0809 01:53:13.108750 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:13.108756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:13.108818 (+ 62us) server_negotiation.cc:407] Connection header received
0809 01:53:13.108926 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:13.108928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:13.108978 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:13.109053 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:13.109508 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.110070 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.110931 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.111150 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.111226 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:13.114185 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:44001: BlockingRecv error: recv got EOF from 127.3.220.3:44001 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 09 01:53:13 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:13 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:13.725917 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:13.719843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:49767)
0809 01:53:13.720125 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:13.720139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:13.720229 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:13.720521 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:13.720523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:13.720771 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:13.720945 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:13.720952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.721991 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.721997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:13.722805 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:13.722815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.723284 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.723288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:13.723347 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:13.723890 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:13.723911 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:13.725803 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
W20260809 01:53:13.726097 17212 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:13.719990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:49767 (local address 127.3.220.62:45159)
0809 01:53:13.720281 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:13.720284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:13.720303 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:53:13.720345 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:13.720347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:13.720393 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:13.720485 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:13.721076 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.721858 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.722918 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.723180 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.723274 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:13.725949 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:49767: BlockingRecv error: recv got EOF from 127.3.220.1:49767 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":127,"threads_started":1}
Aug 09 01:53:13 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:13 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:13.918756 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:13.913148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:40881)
0809 01:53:13.913454 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:13.913467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:13.913635 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:13.913909 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:13.913911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:13.914097 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:13.914193 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:13.914198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.915003 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.915006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:13.915562 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:13.915569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.916029 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.916032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:13.916097 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:13.916627 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:13.916639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:13.918637 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":128,"threads_started":1}
W20260809 01:53:13.918889 17212 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:13.913228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:40881 (local address 127.3.220.62:45159)
0809 01:53:13.913412 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:13.913420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:13.913600 (+ 180us) server_negotiation.cc:407] Connection header received
0809 01:53:13.913734 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:13.913737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:13.913790 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:13.913873 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:13.914313 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.914895 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.915673 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:13.915922 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:13.916013 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:13.918766 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:40881: BlockingRecv error: recv got EOF from 127.3.220.2:40881 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 01:53:14 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:14 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:14.121793 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:14.114930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:36769)
0809 01:53:14.115272 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:14.115293 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:14.115474 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:14.115848 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:14.115852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:14.116134 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:14.116248 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:14.116255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.117167 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.117175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:14.117942 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:14.117954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.118504 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.118509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:14.118623 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:14.119299 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:14.119313 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:14.121576 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
W20260809 01:53:14.121897 17212 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:14.115002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:36769 (local address 127.3.220.62:45159)
0809 01:53:14.115252 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:14.115259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:14.115398 (+ 139us) server_negotiation.cc:407] Connection header received
0809 01:53:14.115604 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:14.115611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:14.115684 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:14.115820 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:14.116401 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.117015 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.118107 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.118364 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.118499 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:14.121744 (+ 3245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:36769: BlockingRecv error: recv got EOF from 127.3.220.3:36769 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Aug 09 01:53:14 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:14 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:14.732621 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:14.726810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:47713)
0809 01:53:14.727153 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:14.727169 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:14.727302 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:14.727650 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:14.727653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:14.727860 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:14.727958 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:14.727963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.728813 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.728815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:14.729447 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:14.729456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.729990 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.729992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:14.730054 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:14.730555 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:14.730566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:14.732443 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
W20260809 01:53:14.732698 17217 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:14.726898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:47713 (local address 127.3.220.62:45159)
0809 01:53:14.727319 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:14.727325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:14.727371 (+ 46us) server_negotiation.cc:407] Connection header received
0809 01:53:14.727416 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:14.727421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:14.727479 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:14.727583 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:14.728097 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.728709 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.729622 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.729889 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.729982 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:14.732573 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:47713: BlockingRecv error: recv got EOF from 127.3.220.1:47713 (error 108)
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":119,"threads_started":1}
Aug 09 01:53:14 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:14 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:14.931209 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:14.919844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:44639)
0809 01:53:14.920186 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:14.920199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:14.920301 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:14.920527 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:14.920530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:14.920730 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:14.920826 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:14.920831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.921688 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.921691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:14.922377 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:14.922387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.922793 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.922796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:14.922857 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:14.928831 (+ 5974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:14.928847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:14.931026 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":94,"threads_started":1}
W20260809 01:53:14.931290 17217 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:14.919919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:44639 (local address 127.3.220.62:45159)
0809 01:53:14.920078 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:14.920082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:14.920267 (+ 185us) server_negotiation.cc:407] Connection header received
0809 01:53:14.920370 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:14.920372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:14.920417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:14.920491 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:14.920926 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.921510 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.922477 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:14.922694 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:14.922776 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:14.931176 (+ 8400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:44639: BlockingRecv error: recv got EOF from 127.3.220.2:44639 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 01:53:15 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:15 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:15.128674 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:15.122869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:40659)
0809 01:53:15.123264 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:15.123276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:15.123430 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:15.123719 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:15.123722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:15.123939 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:15.124032 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:15.124037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.124806 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.124812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:15.125386 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:15.125394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.125911 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.125915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:15.126005 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:15.126558 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:15.126570 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:15.128500 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":126,"threads_started":1}
W20260809 01:53:15.128729 17217 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:15.122932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:40659 (local address 127.3.220.62:45159)
0809 01:53:15.123095 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:15.123101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:15.123369 (+ 268us) server_negotiation.cc:407] Connection header received
0809 01:53:15.123549 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:15.123552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:15.123608 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:15.123691 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:15.124156 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.124694 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.125519 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.125786 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.125907 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:15.128620 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:40659: BlockingRecv error: recv got EOF from 127.3.220.3:40659 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 01:53:15 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:15 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:15.738977 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:15.733503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:41453)
0809 01:53:15.733852 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:15.733866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:15.733959 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:15.734303 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:15.734305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:15.734490 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:15.734590 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:15.734595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.735324 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.735326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:15.735888 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:15.735896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.736302 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.736304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:15.736367 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:15.736859 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:15.736871 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:15.738862 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
W20260809 01:53:15.739171 17230 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:15.733704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:41453 (local address 127.3.220.62:45159)
0809 01:53:15.733926 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:15.733929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:15.733943 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:53:15.734063 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:15.734065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:15.734110 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:15.734210 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:15.734692 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.735231 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.735987 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.736207 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.736280 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:15.738996 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:41453: BlockingRecv error: recv got EOF from 127.3.220.1:41453 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
Aug 09 01:53:15 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:15 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:15.937759 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:15.932038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:50835)
0809 01:53:15.932310 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:15.932323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:15.932424 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:15.932666 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:15.932668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:15.932868 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:15.932967 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:15.932972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.933870 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.933873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:15.934448 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:15.934456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.934878 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.934881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:15.934946 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:15.935513 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:15.935525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:15.937635 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260809 01:53:15.937814 17230 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:15.932091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:50835 (local address 127.3.220.62:45159)
0809 01:53:15.932240 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:15.932244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:15.932395 (+ 151us) server_negotiation.cc:407] Connection header received
0809 01:53:15.932507 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:15.932509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:15.932556 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:15.932627 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:15.933067 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.933752 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.934548 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:15.934755 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:15.934854 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:15.937713 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:50835: BlockingRecv error: recv got EOF from 127.3.220.2:50835 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 01:53:16 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:16 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:16.135037 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:16.129643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:59745)
0809 01:53:16.129923 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:16.129935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:16.130036 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:16.130351 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:16.130355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:16.130615 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:16.130734 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:16.130741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.131517 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.131519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:16.132045 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:16.132052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.132471 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.132473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:16.132537 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:16.133039 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:16.133051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:16.134869 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":121,"threads_started":1}
W20260809 01:53:16.135108 17230 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:16.129727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:59745 (local address 127.3.220.62:45159)
0809 01:53:16.129875 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:16.129879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:16.130007 (+ 128us) server_negotiation.cc:407] Connection header received
0809 01:53:16.130122 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:16.130125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:16.130213 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:16.130281 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:16.130887 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.131423 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.132143 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.132366 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.132445 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:16.134996 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:59745: BlockingRecv error: recv got EOF from 127.3.220.3:59745 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 01:53:16 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:16 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:16.745824 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:16.739923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:36185)
0809 01:53:16.740214 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:16.740228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:16.740351 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:16.740692 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:16.740699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:16.740965 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:16.741099 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:16.741106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.742126 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.742128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:16.742684 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:16.742692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.743129 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.743131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:16.743194 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:16.743752 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:16.743764 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:16.745702 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
W20260809 01:53:16.746042 17236 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:16.740009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:36185 (local address 127.3.220.62:45159)
0809 01:53:16.740276 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:16.740282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:16.740299 (+ 17us) server_negotiation.cc:407] Connection header received
0809 01:53:16.740457 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:16.740478 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:16.740544 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:16.740659 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:16.741270 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.742026 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.742804 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.743030 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.743116 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:16.745861 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:36185: BlockingRecv error: recv got EOF from 127.3.220.1:36185 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
Aug 09 01:53:16 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:16 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:16.944416 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:16.938729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:51635)
0809 01:53:16.939020 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:16.939035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:16.939171 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:16.939466 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:16.939470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:16.939688 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:16.939783 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:16.939789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.940552 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.940557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:16.941122 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:16.941130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.941615 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.941619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:16.941708 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:16.942333 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:16.942346 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:16.944299 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
W20260809 01:53:16.944592 17236 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:16.938771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:51635 (local address 127.3.220.62:45159)
0809 01:53:16.938958 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:16.938963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:16.939120 (+ 157us) server_negotiation.cc:407] Connection header received
0809 01:53:16.939286 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:16.939289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:16.939346 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:16.939435 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:16.939905 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.940446 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.941241 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:16.941477 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:16.941615 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:16.944434 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:51635: BlockingRecv error: recv got EOF from 127.3.220.2:51635 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 01:53:17 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:17 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:17.142059 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:17.135970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:32949)
0809 01:53:17.136325 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:17.136347 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:17.136482 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:17.136775 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:17.136778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:17.136997 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:17.137087 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:17.137093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.137932 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.137937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:17.138754 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:17.138766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.139391 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.139395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:17.139486 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:17.140004 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:17.140015 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:17.141944 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
W20260809 01:53:17.142238 17236 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:17.136112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:32949 (local address 127.3.220.62:45159)
0809 01:53:17.136278 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:17.136283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:17.136425 (+ 142us) server_negotiation.cc:407] Connection header received
0809 01:53:17.136597 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:17.136600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:17.136656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:17.136749 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:17.137212 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.137822 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.138881 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.139238 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.139381 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:17.142066 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:32949: BlockingRecv error: recv got EOF from 127.3.220.3:32949 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 01:53:17 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:17 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:17.752421 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:17.746748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:48081)
0809 01:53:17.747177 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:17.747194 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:17.747324 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:17.747705 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:17.747709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:17.747947 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:17.748067 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:17.748074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.748812 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.748814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:17.749386 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:17.749393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.749828 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.749831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:17.749897 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:17.750394 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:17.750406 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:17.752246 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":108,"threads_started":1}
W20260809 01:53:17.752612 17243 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:17.746914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:48081 (local address 127.3.220.62:45159)
0809 01:53:17.747182 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:17.747187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:17.747429 (+ 242us) server_negotiation.cc:407] Connection header received
0809 01:53:17.747472 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:17.747474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:17.747544 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:17.747642 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:17.748175 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.748720 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.749488 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.749732 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.749809 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:17.752380 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:48081: BlockingRecv error: recv got EOF from 127.3.220.1:48081 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
Aug 09 01:53:17 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:17 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:17.950863 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:17.945296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:60421)
0809 01:53:17.945607 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:17.945621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:17.945719 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:17.945945 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:17.945947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:17.946130 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:17.946228 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:17.946233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.946979 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.946982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:17.947527 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:17.947534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.947974 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.947976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:17.948039 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:17.948527 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:17.948539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:17.950681 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
W20260809 01:53:17.950904 17243 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:17.945387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:60421 (local address 127.3.220.62:45159)
0809 01:53:17.945552 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:17.945556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:17.945672 (+ 116us) server_negotiation.cc:407] Connection header received
0809 01:53:17.945786 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:17.945788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:17.945837 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:17.945902 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:17.946338 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.946885 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.947626 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:17.947875 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:17.947952 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:17.950804 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:60421: BlockingRecv error: recv got EOF from 127.3.220.2:60421 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 01:53:18 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:18 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:18.148185 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:18.142895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:37839)
0809 01:53:18.143198 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:18.143211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:18.143320 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:18.143549 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:18.143552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:18.143739 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:18.143841 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:18.143846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.144598 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.144600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:18.145176 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:18.145184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.145678 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.145681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:18.145746 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:18.146238 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:18.146249 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:18.148043 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":127,"threads_started":1}
W20260809 01:53:18.148268 17243 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:18.143031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:37839 (local address 127.3.220.62:45159)
0809 01:53:18.143176 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:18.143180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:18.143260 (+ 80us) server_negotiation.cc:407] Connection header received
0809 01:53:18.143388 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:18.143390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:18.143436 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:18.143502 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:18.143944 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.144496 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.145278 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.145502 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.145665 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:18.148176 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:37839: BlockingRecv error: recv got EOF from 127.3.220.3:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 01:53:18 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:18 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:18.759070 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:18.753272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:57689)
0809 01:53:18.753648 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:18.753665 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:18.753794 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:18.754094 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:18.754097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:18.754350 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:18.754457 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:18.754463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.755251 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.755257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:18.755835 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:18.755843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.756328 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.756332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:18.756418 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:18.756925 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:18.756936 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:18.758894 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":89,"threads_started":1}
W20260809 01:53:18.759146 17249 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:18.753320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:57689 (local address 127.3.220.62:45159)
0809 01:53:18.753617 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:18.753621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:18.753754 (+ 133us) server_negotiation.cc:407] Connection header received
0809 01:53:18.753900 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:18.753904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:18.753963 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:18.754066 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:18.754582 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.755138 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.755967 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.756204 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.756317 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:18.759020 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:57689: BlockingRecv error: recv got EOF from 127.3.220.1:57689 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":63,"threads_started":1}
Aug 09 01:53:18 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:18 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:18.957432 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:18.951661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:42331)
0809 01:53:18.952101 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:18.952114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:18.952254 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:18.952535 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:18.952539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:18.952754 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:18.952851 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:18.952857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.953670 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.953675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:18.954242 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:18.954250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.954750 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.954753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:18.954845 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:18.955365 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:18.955376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:18.957286 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":123,"threads_started":1}
W20260809 01:53:18.957515 17249 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:18.951749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:42331 (local address 127.3.220.62:45159)
0809 01:53:18.951925 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:18.951930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:18.952196 (+ 266us) server_negotiation.cc:407] Connection header received
0809 01:53:18.952369 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:18.952378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:18.952423 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:18.952506 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:18.952976 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.953514 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.954387 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:18.954635 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:18.954743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:18.957423 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:42331: BlockingRecv error: recv got EOF from 127.3.220.2:42331 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 01:53:19 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:19 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:19.154834 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:19.149059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:45101)
0809 01:53:19.149379 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:19.149397 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:19.149518 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:19.149868 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:19.149872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:19.150118 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:19.150244 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:19.150252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.151039 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.151043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:19.151618 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:19.151626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.152114 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.152118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:19.152206 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:19.152777 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:19.152803 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:19.154718 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":102,"threads_started":1}
W20260809 01:53:19.154985 17249 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:19.149258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:45101 (local address 127.3.220.62:45159)
0809 01:53:19.149389 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:19.149396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:19.149465 (+ 69us) server_negotiation.cc:407] Connection header received
0809 01:53:19.149663 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:19.149668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:19.149734 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:19.149842 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:19.150371 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.150923 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.151749 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.151994 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.152101 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:19.154860 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:45101: BlockingRecv error: recv got EOF from 127.3.220.3:45101 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 01:53:19 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:19 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:19.766172 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:19.759880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:58373)
0809 01:53:19.760336 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:19.760351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:19.760460 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:19.760721 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:19.760723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:19.760910 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:19.761012 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:19.761017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.761762 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.761765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:19.762337 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:19.762345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.762779 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.762782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:19.762851 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:19.763527 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:19.763550 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:19.766037 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":97,"threads_started":1}
W20260809 01:53:19.766304 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:19.760118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:58373 (local address 127.3.220.62:45159)
0809 01:53:19.760365 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:19.760369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:19.760428 (+ 59us) server_negotiation.cc:407] Connection header received
0809 01:53:19.760546 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:19.760548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:19.760592 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:19.760671 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:19.761116 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.761671 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.762437 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.762654 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.762739 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:19.766151 (+ 3412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:58373: BlockingRecv error: recv got EOF from 127.3.220.1:58373 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
Aug 09 01:53:19 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:19 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:19.968677 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:19.958348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:36749)
0809 01:53:19.958737 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:19.958753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:19.958905 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:19.959235 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:19.959238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:19.959469 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:19.959562 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:19.959568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.960408 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.960415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:19.961005 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:19.961013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.961593 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.961597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:19.961703 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:19.966311 (+ 4608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:19.966324 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:19.968491 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":95,"threads_started":1}
W20260809 01:53:19.968745 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:19.958514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:36749 (local address 127.3.220.62:45159)
0809 01:53:19.958703 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:19.958708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:19.958840 (+ 132us) server_negotiation.cc:407] Connection header received
0809 01:53:19.959052 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:19.959056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:19.959106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:19.959201 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:19.959694 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.960286 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.961144 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:19.961437 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:19.961575 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:19.968636 (+ 7061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:36749: BlockingRecv error: recv got EOF from 127.3.220.2:36749 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 09 01:53:20 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:20 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:20.162128 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:20.155744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:47469)
0809 01:53:20.156087 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:20.156105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:20.156273 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:20.156588 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:20.156592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:20.156828 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:20.156930 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:20.156936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.157890 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.157895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:20.158549 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:20.158557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.159061 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.159065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:20.159167 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:20.159803 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:20.159817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:20.162015 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
W20260809 01:53:20.162304 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:20.155905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:47469 (local address 127.3.220.62:45159)
0809 01:53:20.156081 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:20.156086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:20.156205 (+ 119us) server_negotiation.cc:407] Connection header received
0809 01:53:20.156405 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:20.156408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:20.156459 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:20.156562 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:20.157064 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.157773 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.158686 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.158935 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.159059 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:20.162164 (+ 3105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:47469: BlockingRecv error: recv got EOF from 127.3.220.3:47469 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 01:53:20 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:20 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:20.772871 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:20.767096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:52997)
0809 01:53:20.767411 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:20.767427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:20.767537 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:20.767814 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:20.767816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:20.767994 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:20.768094 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:20.768099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.768956 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.768959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:20.769622 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:20.769633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.770103 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.770106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:20.770190 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:20.770772 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:20.770784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:20.772744 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
W20260809 01:53:20.773061 17259 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:20.767197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:52997 (local address 127.3.220.62:45159)
0809 01:53:20.767476 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:20.767480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:20.767495 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:53:20.767645 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:20.767647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:20.767696 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:20.767780 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:20.768234 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.768856 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.769755 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.770002 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.770090 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:20.772887 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:52997: BlockingRecv error: recv got EOF from 127.3.220.1:52997 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":64,"threads_started":1}
Aug 09 01:53:20 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:20 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:20.975493 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:20.969664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:37455)
0809 01:53:20.969975 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:20.969996 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:20.970137 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:20.970438 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:20.970441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:20.970677 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:20.970775 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:20.970780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.971562 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.971567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:20.972146 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:20.972156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.972631 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.972635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:20.972733 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:20.973360 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:20.973373 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:20.975287 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
W20260809 01:53:20.975531 17259 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:20.969725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:37455 (local address 127.3.220.62:45159)
0809 01:53:20.969912 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:20.969918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:20.970080 (+ 162us) server_negotiation.cc:407] Connection header received
0809 01:53:20.970276 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:20.970279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:20.970324 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:20.970412 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:20.970901 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.971446 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.972275 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:20.972511 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:20.972626 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:20.975425 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:37455: BlockingRecv error: recv got EOF from 127.3.220.2:37455 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 09 01:53:21 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:21 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:21.169335 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:21.163045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:48889)
0809 01:53:21.163450 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:21.163471 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:21.163628 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:21.163953 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:21.163956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:21.164229 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:21.164330 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:21.164336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.165190 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.165197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:21.165876 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:21.165885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.166424 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.166428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:21.166525 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:21.167099 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:21.167145 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:21.169149 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":119,"threads_started":1}
W20260809 01:53:21.169394 17259 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:21.163244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:48889 (local address 127.3.220.62:45159)
0809 01:53:21.163425 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:21.163430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:21.163560 (+ 130us) server_negotiation.cc:407] Connection header received
0809 01:53:21.163746 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:21.163750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:21.163812 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:21.163929 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:21.164468 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.165059 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.166015 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.166294 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.166421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:21.169283 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:48889: BlockingRecv error: recv got EOF from 127.3.220.3:48889 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 01:53:21 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:21 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:21.779754 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:21.773689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:51339)
0809 01:53:21.773986 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:21.774012 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:21.774104 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:21.774350 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:21.774352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:21.774533 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:21.774628 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:21.774634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.775451 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.775453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:21.776022 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:21.776030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.776591 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.776598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:21.776717 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:21.777349 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:21.777368 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:21.779554 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
W20260809 01:53:21.779844 17268 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:21.773752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:51339 (local address 127.3.220.62:45159)
0809 01:53:21.774106 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:21.774110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:21.774152 (+ 42us) server_negotiation.cc:407] Connection header received
0809 01:53:21.774190 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:21.774193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:21.774238 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:21.774319 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:21.774736 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.775354 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.776165 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.776421 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.776562 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:21.779679 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:51339: BlockingRecv error: recv got EOF from 127.3.220.1:51339 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":77,"threads_started":1}
Aug 09 01:53:21 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:21 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:21.982514 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:21.976593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:50693)
0809 01:53:21.976927 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:21.976940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:21.977048 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:21.977308 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:21.977310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:21.977500 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:21.977622 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:21.977627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.978430 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.978433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:21.979088 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:21.979097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.979658 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.979662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:21.979753 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:21.980357 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:21.980369 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:21.982384 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
W20260809 01:53:21.982687 17268 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:21.976675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:50693 (local address 127.3.220.62:45159)
0809 01:53:21.976857 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:21.976861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:21.977014 (+ 153us) server_negotiation.cc:407] Connection header received
0809 01:53:21.977132 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:21.977134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:21.977182 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:21.977265 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:21.977725 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.978325 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.979214 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:21.979525 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:21.979655 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:21.982528 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:50693: BlockingRecv error: recv got EOF from 127.3.220.2:50693 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 09 01:53:22 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:22 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.3@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:22.175981 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:22.170297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.3:56167)
0809 01:53:22.170581 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:22.170594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:22.170691 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:22.170974 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:22.170977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:22.171174 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:22.171278 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:22.171284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.172031 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.172035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:22.172686 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:22.172695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.173133 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.173137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:22.173217 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:22.173810 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:22.173823 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:22.175853 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":112,"threads_started":1}
W20260809 01:53:22.176177 17268 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:22.170393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:56167 (local address 127.3.220.62:45159)
0809 01:53:22.170527 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:22.170530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:22.170662 (+ 132us) server_negotiation.cc:407] Connection header received
0809 01:53:22.170754 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:22.170756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:22.170802 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:22.170874 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:22.171399 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.171918 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.172803 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.173015 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.173092 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:22.175990 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.3:56167: BlockingRecv error: recv got EOF from 127.3.220.3:56167 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 01:53:22 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:22 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.1@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:22.787971 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:22.781480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.1:60695)
0809 01:53:22.781758 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:22.781774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:22.781898 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:22.782352 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:22.782355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:22.782610 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:22.782744 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:22.782751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.783704 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.783709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:22.784294 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:22.784303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.784789 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.784793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:22.784877 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:22.785440 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:22.785454 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:22.787796 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
W20260809 01:53:22.788044 17272 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:22.781867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:60695 (local address 127.3.220.62:45159)
0809 01:53:22.782094 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:22.782100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:22.782117 (+ 17us) server_negotiation.cc:407] Connection header received
0809 01:53:22.782164 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:22.782169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:22.782232 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:22.782329 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:22.782869 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.783594 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.784421 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.784669 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.784791 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:22.787913 (+ 3122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.1:60695: BlockingRecv error: recv got EOF from 127.3.220.1:60695 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
Aug 09 01:53:22 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 01:53:22 dist-test-slave-gkw7 krb5kdc[15768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.220.2@KRBTEST.COM for oryx/127.3.220.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 01:53:22.989481 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:22.983593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:45159 (local address 127.3.220.2:59505)
0809 01:53:22.983974 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:22.983992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:22.984151 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:22.984450 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:22.984453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:22.984725 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:22.984850 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:22.984856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.985783 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.985787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:22.986366 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:22.986375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.986868 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.986871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:22.986965 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:22.987470 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:22.987482 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:22.989361 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.220.62:45159: Server oryx/127.3.220.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":158,"threads_started":1}
W20260809 01:53:22.989708 17272 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:22.983801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:59505 (local address 127.3.220.62:45159)
0809 01:53:22.983977 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:22.983984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:22.984093 (+ 109us) server_negotiation.cc:407] Connection header received
0809 01:53:22.984273 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:22.984276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:22.984329 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:22.984425 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:22.984973 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.985674 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.986491 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:22.986746 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:22.986860 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:22.989502 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.220.2:59505: BlockingRecv error: recv got EOF from 127.3.220.2:59505 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 01:53:23.066558 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 16602
I20260809 01:53:23.089156 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 16804
I20260809 01:53:23.107928 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 16990
I20260809 01:53:23.127722 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 16505
2026-08-09T01:53:23Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18373 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T01:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:53:23Z Disabled control of system clock
I20260809 01:53:23.164379 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44647
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:44647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 01:53:23.278067 17281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:23.278450 17281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:23.278507 17281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:23.282708 17281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:53:23.282768 17281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:23.282783 17281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:23.282799 17281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:53:23.282850 17281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:53:23.287631 17281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:44647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:23.288684 17281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:23.290143 17281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:23.299141 17288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:23.299162 17290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:23.299240 17281 server_base.cc:1034] running on GCE node
W20260809 01:53:23.299259 17287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:23.300050 17281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:23.301139 17281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:23.302335 17281 hybrid_clock.cc:648] HybridClock initialized: now 1786240403302240 us; error 106 us; skew 500 ppm
I20260809 01:53:23.304416 17281 webserver.cc:460] Webserver started at http://127.3.220.62:34019/ using document root <none> and password file <none>
I20260809 01:53:23.305083 17281 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:23.305150 17281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:23.305320 17281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:23.307374 17281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "b0d3aecdb301431cad49fff8aadfda58"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "720e48389a1bfcf4dcc059e6ddf36741"
server_key_iv: "b8c9f6aa71c8c1ed3882e8aa35df215d"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.307901 17281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "b0d3aecdb301431cad49fff8aadfda58"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "720e48389a1bfcf4dcc059e6ddf36741"
server_key_iv: "b8c9f6aa71c8c1ed3882e8aa35df215d"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.311539 17281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:53:23.313907 17297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.315130 17281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:53:23.315267 17281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "b0d3aecdb301431cad49fff8aadfda58"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "720e48389a1bfcf4dcc059e6ddf36741"
server_key_iv: "b8c9f6aa71c8c1ed3882e8aa35df215d"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.315382 17281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:23.333976 17281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:23.334798 17281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:23.335049 17281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:23.342931 17281 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:44647
I20260809 01:53:23.342935 17368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:44647 every 8 connection(s)
I20260809 01:53:23.343987 17281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:53:23.347003 17369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:23.350808 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 17281
I20260809 01:53:23.350916 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:53:23.351261 3952 external_mini_cluster.cc:1428] Setting key 58246212b031d6def6ea73ccf7d94d6b
I20260809 01:53:23.353017 17369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58: Bootstrap starting.
I20260809 01:53:23.355456 17369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:23.356209 17369 log.cc:824] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:23.358379 17369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58: No bootstrap required, opened a new log
I20260809 01:53:23.360725 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:23.352849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.3.220.62:44647)
0809 01:53:23.353471 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:23.353487 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:23.353524 (+ 37us) server_negotiation.cc:407] Connection header received
0809 01:53:23.354227 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:23.354245 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:23.354653 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:23.354961 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:23.355316 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.356322 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.357485 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.358009 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.358129 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:23.359011 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:23.359016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:23.359028 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:53:23.359427 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:23.359481 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:53:23.359500 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:23.359577 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:23.359752 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 01:53:23.360032 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":143,"threads_started":1}
I20260809 01:53:23.362207 17369 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d3aecdb301431cad49fff8aadfda58" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44647 } }
I20260809 01:53:23.362416 17369 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:23.362488 17369 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d3aecdb301431cad49fff8aadfda58, State: Initialized, Role: FOLLOWER
I20260809 01:53:23.362984 17369 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d3aecdb301431cad49fff8aadfda58" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44647 } }
I20260809 01:53:23.363161 17369 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:53:23.363229 17369 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:53:23.363325 17369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:23.364205 17369 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d3aecdb301431cad49fff8aadfda58" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44647 } }
I20260809 01:53:23.364543 17369 leader_election.cc:302] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0d3aecdb301431cad49fff8aadfda58; no voters:
I20260809 01:53:23.364861 17369 leader_election.cc:288] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:53:23.364972 17380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:23.365231 17380 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [term 1 LEADER]: Becoming Leader. State: Replica: b0d3aecdb301431cad49fff8aadfda58, State: Running, Role: LEADER
I20260809 01:53:23.365638 17380 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d3aecdb301431cad49fff8aadfda58" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44647 } }
I20260809 01:53:23.366297 17369 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:53:23.367396 17381 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0d3aecdb301431cad49fff8aadfda58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d3aecdb301431cad49fff8aadfda58" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44647 } } }
I20260809 01:53:23.367540 17381 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:23.367925 17393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:53:23.368261 17384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0d3aecdb301431cad49fff8aadfda58. Latest consensus state: current_term: 1 leader_uuid: "b0d3aecdb301431cad49fff8aadfda58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d3aecdb301431cad49fff8aadfda58" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44647 } } }
I20260809 01:53:23.368374 17384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:23.370630 17393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:53:23.375965 17393 catalog_manager.cc:1347] Generated new cluster ID: ddc799851c6943db9d18a4018a74bab1
I20260809 01:53:23.376036 17393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:53:23.397513 17393 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:53:23.398836 17393 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:53:23.404821 17393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0d3aecdb301431cad49fff8aadfda58: Generated new TSK 0
I20260809 01:53:23.405663 17393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:53:23.418388 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44647
--builtin_ntp_servers=127.3.220.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 01:53:23.536314 17407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:23.536614 17407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:23.536684 17407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:23.540755 17407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:23.540845 17407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:23.540937 17407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:53:23.545847 17407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:23.547012 17407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:23.548472 17407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:23.558743 17419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:23.558723 17413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:23.558719 17414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:23.559196 17407 server_base.cc:1034] running on GCE node
I20260809 01:53:23.559584 17407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:23.560211 17407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:23.561378 17407 hybrid_clock.cc:648] HybridClock initialized: now 1786240403561344 us; error 45 us; skew 500 ppm
I20260809 01:53:23.563328 17407 webserver.cc:460] Webserver started at http://127.3.220.1:45217/ using document root <none> and password file <none>
I20260809 01:53:23.563923 17407 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:23.563988 17407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:23.564265 17407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:23.566425 17407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "d1795ea8346d4112bc71b618021d5cbf"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "8469ec18a99b42c4cc3b624dcfd3204b"
server_key_iv: "7ffeb3a85374d2b0ac82ec790b0e4781"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.567070 17407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "d1795ea8346d4112bc71b618021d5cbf"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "8469ec18a99b42c4cc3b624dcfd3204b"
server_key_iv: "7ffeb3a85374d2b0ac82ec790b0e4781"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.571058 17407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:53:23.574095 17428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.575443 17407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:53:23.575610 17407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "d1795ea8346d4112bc71b618021d5cbf"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "8469ec18a99b42c4cc3b624dcfd3204b"
server_key_iv: "7ffeb3a85374d2b0ac82ec790b0e4781"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.575753 17407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:23.587865 17407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:23.588586 17407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:23.588845 17407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:23.589483 17407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:23.590613 17407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:23.590713 17407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.590811 17407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:23.590868 17407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.600833 17407 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:45515
I20260809 01:53:23.600864 17591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:45515 every 8 connection(s)
I20260809 01:53:23.602065 17407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:53:23.606365 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 17407
I20260809 01:53:23.606462 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:53:23.606767 3952 external_mini_cluster.cc:1428] Setting key ae43c63283b168eee6114867e5f90a61
I20260809 01:53:23.609071 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44647
--builtin_ntp_servers=127.3.220.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 01:53:23.612828 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:23.604181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:40541 (local address 127.3.220.62:44647)
0809 01:53:23.604373 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:23.604377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:23.605632 (+ 1255us) server_negotiation.cc:407] Connection header received
0809 01:53:23.607316 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:23.607320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:23.607398 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:23.607498 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:23.608774 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.609464 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.610701 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.610947 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.611066 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:23.612294 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:23.612297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:23.612300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:53:23.612314 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:23.612345 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:53:23.612353 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:23.612436 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:23.612604 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 01:53:23.612676 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 01:53:23.613889 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:23.604658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44647 (local address 127.3.220.1:40541)
0809 01:53:23.605422 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:23.605467 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:23.607041 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:23.607756 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:23.607766 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:23.607880 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:23.608607 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:23.608624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.609635 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.609648 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:23.610554 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:23.610564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.611118 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.611122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:23.611314 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:23.612002 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:53:23.612004 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:23.612049 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:53:23.612122 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:23.612438 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:23.612451 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:23.612459 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 01:53:23.612605 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 01:53:23.612947 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":120,"threads_started":1}
I20260809 01:53:23.615285 17592 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44647
I20260809 01:53:23.615592 17592 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:23.616346 17592 heartbeater.cc:511] Master 127.3.220.62:44647 requested a full tablet report, sending...
I20260809 01:53:23.618412 17318 ts_manager.cc:194] Registered new tserver with Master: d1795ea8346d4112bc71b618021d5cbf (127.3.220.1:45515)
I20260809 01:53:23.620221 17318 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.1:40541
W20260809 01:53:23.726476 17595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:23.726737 17595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:23.726773 17595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:23.730645 17595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:23.730695 17595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:23.730746 17595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:53:23.735280 17595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:23.736382 17595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:23.737757 17595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:23.747956 17605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:23.747963 17602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:23.747963 17601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:23.748028 17595 server_base.cc:1034] running on GCE node
I20260809 01:53:23.748684 17595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:23.749308 17595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:23.750516 17595 hybrid_clock.cc:648] HybridClock initialized: now 1786240403750482 us; error 56 us; skew 500 ppm
I20260809 01:53:23.752530 17595 webserver.cc:460] Webserver started at http://127.3.220.2:45145/ using document root <none> and password file <none>
I20260809 01:53:23.753108 17595 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:23.753214 17595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:23.753454 17595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:23.755761 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "5c5f22725ae442d1a104f123535f9a9f"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "d8bd0f21c0ec6351d95ad55cdf8c637e"
server_key_iv: "41953af0a0d9c763e4da3ba837d8cdc2"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.756357 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "5c5f22725ae442d1a104f123535f9a9f"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "d8bd0f21c0ec6351d95ad55cdf8c637e"
server_key_iv: "41953af0a0d9c763e4da3ba837d8cdc2"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.760237 17595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 01:53:23.762840 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.764067 17595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:53:23.764213 17595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "5c5f22725ae442d1a104f123535f9a9f"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "d8bd0f21c0ec6351d95ad55cdf8c637e"
server_key_iv: "41953af0a0d9c763e4da3ba837d8cdc2"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.764346 17595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:23.783563 17595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:23.784286 17595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:23.784484 17595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:23.785079 17595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:23.786087 17595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:23.786160 17595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.786237 17595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:23.786291 17595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.795043 17595 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:36825
I20260809 01:53:23.795068 17770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:36825 every 8 connection(s)
I20260809 01:53:23.795969 17595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:53:23.796991 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 17595
I20260809 01:53:23.797116 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:53:23.797448 3952 external_mini_cluster.cc:1428] Setting key f297250beac6497bf370ff76f5a64954
I20260809 01:53:23.799640 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44647
--builtin_ntp_servers=127.3.220.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 01:53:23.805711 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:23.798034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:59743 (local address 127.3.220.62:44647)
0809 01:53:23.798380 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:23.798385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:23.799389 (+ 1004us) server_negotiation.cc:407] Connection header received
0809 01:53:23.800889 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:23.800892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:23.800946 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:23.801056 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:23.802028 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.802562 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.803709 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.803937 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.804050 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:23.805165 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:23.805168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:23.805171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:53:23.805191 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:23.805211 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:53:23.805219 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:23.805314 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:23.805474 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 01:53:23.805564 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260809 01:53:23.806547 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:23.798429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44647 (local address 127.3.220.2:59743)
0809 01:53:23.799239 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:23.799281 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:23.800665 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:23.801235 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:23.801242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:23.801325 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:23.801881 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:23.801895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.802695 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.802703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:23.803581 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:23.803591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.804095 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.804099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:23.804262 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:23.804890 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:53:23.804892 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:23.804930 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:53:23.804991 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:23.805318 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:23.805330 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:23.805337 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 01:53:23.805474 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 01:53:23.805772 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":124,"threads_started":1}
I20260809 01:53:23.807869 17771 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44647
I20260809 01:53:23.808109 17771 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:23.808625 17771 heartbeater.cc:511] Master 127.3.220.62:44647 requested a full tablet report, sending...
I20260809 01:53:23.809806 17318 ts_manager.cc:194] Registered new tserver with Master: 5c5f22725ae442d1a104f123535f9a9f (127.3.220.2:36825)
I20260809 01:53:23.810695 17318 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.2:59743
W20260809 01:53:23.917583 17775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:23.917860 17775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:23.917896 17775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:23.921981 17775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:23.922042 17775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:23.922096 17775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:53:23.927728 17775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:23.929208 17775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:23.930751 17775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:23.940727 17783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:23.940727 17791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:23.940773 17785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:23.941208 17775 server_base.cc:1034] running on GCE node
I20260809 01:53:23.941720 17775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:23.942448 17775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:23.943646 17775 hybrid_clock.cc:648] HybridClock initialized: now 1786240403943612 us; error 49 us; skew 500 ppm
I20260809 01:53:23.945501 17775 webserver.cc:460] Webserver started at http://127.3.220.3:43003/ using document root <none> and password file <none>
I20260809 01:53:23.946151 17775 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:23.946244 17775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:23.946496 17775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:23.948549 17775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "4721e62076c8498b96e8d3b2c7a3fae3"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "e6a4581c57a26bcb4217f70fe411a058"
server_key_iv: "b98792ee27a58d464f8f0bd415e6ea06"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.949100 17775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "4721e62076c8498b96e8d3b2c7a3fae3"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "e6a4581c57a26bcb4217f70fe411a058"
server_key_iv: "b98792ee27a58d464f8f0bd415e6ea06"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.952813 17775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 01:53:23.955435 17800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.956616 17775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:53:23.956758 17775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "4721e62076c8498b96e8d3b2c7a3fae3"
format_stamp: "Formatted at 2026-08-09 01:53:23 on dist-test-slave-gkw7"
server_key: "e6a4581c57a26bcb4217f70fe411a058"
server_key_iv: "b98792ee27a58d464f8f0bd415e6ea06"
server_key_version: "encryptionkey@0"
I20260809 01:53:23.956874 17775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:23.972431 17775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:23.973096 17775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:23.973299 17775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:23.973950 17775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:23.974905 17775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:23.974977 17775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.975050 17775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:23.975095 17775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:23.984202 17775 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:36307
I20260809 01:53:23.984269 17954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:36307 every 8 connection(s)
I20260809 01:53:23.985155 17775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:53:23.987103 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 17775
I20260809 01:53:23.987251 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:53:23.987600 3952 external_mini_cluster.cc:1428] Setting key cc8e72367d8841e1683ddd25ce3b8a72
I20260809 01:53:23.994927 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:23.987028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:39385 (local address 127.3.220.62:44647)
0809 01:53:23.987165 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:23.987172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:23.988637 (+ 1465us) server_negotiation.cc:407] Connection header received
0809 01:53:23.990407 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:23.990410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:23.990459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:23.990528 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:23.991398 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.991920 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.993020 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.993252 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.993335 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:23.994512 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:23.994515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:23.994517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:53:23.994529 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:23.994548 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:53:23.994556 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:23.994614 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:23.994758 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 01:53:23.994817 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:53:23.995813 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:23.987587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44647 (local address 127.3.220.3:39385)
0809 01:53:23.988470 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:23.988528 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:23.990208 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:23.990691 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:23.990698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:23.990779 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:23.991272 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:23.991285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.992030 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.992033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:23.992906 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:23.992915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:23.993387 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:23.993390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:23.993574 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:23.994251 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:53:23.994253 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:23.994293 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:53:23.994355 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:23.994630 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:23.994637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:23.994641 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:53:23.994736 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 01:53:23.994997 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":155,"threads_started":1}
I20260809 01:53:23.997004 17955 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44647
I20260809 01:53:23.997251 17955 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:23.997838 17955 heartbeater.cc:511] Master 127.3.220.62:44647 requested a full tablet report, sending...
I20260809 01:53:23.998906 17318 ts_manager.cc:194] Registered new tserver with Master: 4721e62076c8498b96e8d3b2c7a3fae3 (127.3.220.3:36307)
I20260809 01:53:23.999439 17318 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.3:39385
I20260809 01:53:24.001813 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 01:53:24.004487 17378 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:24.003677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.3.220.62:44647)
0809 01:53:24.003803 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:24.003806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:24.003960 (+ 154us) server_negotiation.cc:407] Connection header received
0809 01:53:24.004054 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:24.004056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:24.004091 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:24.004157 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:24.004395 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50464: BlockingRecv error: recv got EOF from 127.0.0.1:50464 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 01:53:24.004748 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 17407
I20260809 01:53:24.020560 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 17595
I20260809 01:53:24.035085 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 17775
I20260809 01:53:24.059765 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 17281
2026-08-09T01:53:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (932 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T01:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:53:24Z Disabled control of system clock
I20260809 01:53:24.096279 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:32935
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 01:53:24.210016 17970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:24.210286 17970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:24.210325 17970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:24.214679 17970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:53:24.214735 17970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:24.214749 17970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:24.214766 17970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:53:24.214823 17970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:53:24.220118 17970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:32935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:24.221218 17970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:24.222683 17970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:24.231480 17976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:24.231549 17970 server_base.cc:1034] running on GCE node
W20260809 01:53:24.231604 17981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:24.231496 17977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:24.232328 17970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:24.233423 17970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:24.234594 17970 hybrid_clock.cc:648] HybridClock initialized: now 1786240404234568 us; error 39 us; skew 500 ppm
I20260809 01:53:24.236517 17970 webserver.cc:460] Webserver started at http://127.3.220.62:40607/ using document root <none> and password file <none>
I20260809 01:53:24.237169 17970 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:24.237236 17970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:24.237495 17970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:24.239674 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "5cda928d8b984c479353e42d12eea535"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "22a5c2c92bab50d47009207fda6459fa"
server_key_iv: "e2aa167506936a59dd561a3592767a9b"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.240302 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "5cda928d8b984c479353e42d12eea535"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "22a5c2c92bab50d47009207fda6459fa"
server_key_iv: "e2aa167506936a59dd561a3592767a9b"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.244494 17970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 01:53:24.247040 17992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.248320 17970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:53:24.248473 17970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "5cda928d8b984c479353e42d12eea535"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "22a5c2c92bab50d47009207fda6459fa"
server_key_iv: "e2aa167506936a59dd561a3592767a9b"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.248595 17970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:24.253475 17970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:24.254168 17970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:24.254364 17970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:24.261117 17970 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:32935
I20260809 01:53:24.261122 18062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:32935 every 8 connection(s)
I20260809 01:53:24.262250 17970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:53:24.265587 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:24.271306 18063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535: Bootstrap starting.
I20260809 01:53:24.272435 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 17970
I20260809 01:53:24.272545 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:53:24.272883 3952 external_mini_cluster.cc:1428] Setting key 088fe8e301817afe5a230a55f04e73d0
I20260809 01:53:24.274212 18063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:24.275151 18063 log.cc:824] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:24.277490 18063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535: No bootstrap required, opened a new log
I20260809 01:53:24.278930 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:24.274141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.3.220.62:32935)
0809 01:53:24.274931 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:24.274953 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:24.274998 (+ 45us) server_negotiation.cc:407] Connection header received
0809 01:53:24.275796 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:24.275815 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:24.276186 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:24.276558 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:24.277340 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:24.277345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:24.277359 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:53:24.277371 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:24.277443 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:53:24.277466 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:24.277615 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:24.277757 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 01:53:24.278019 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":152,"threads_started":1}
I20260809 01:53:24.281896 18063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cda928d8b984c479353e42d12eea535" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 32935 } }
I20260809 01:53:24.282109 18063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:24.282142 18063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cda928d8b984c479353e42d12eea535, State: Initialized, Role: FOLLOWER
I20260809 01:53:24.282547 18063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cda928d8b984c479353e42d12eea535" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 32935 } }
I20260809 01:53:24.282668 18063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:53:24.282716 18063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:53:24.282793 18063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:24.283650 18063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cda928d8b984c479353e42d12eea535" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 32935 } }
I20260809 01:53:24.283949 18063 leader_election.cc:302] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cda928d8b984c479353e42d12eea535; no voters:
I20260809 01:53:24.284192 18063 leader_election.cc:288] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:53:24.284374 18068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:24.284600 18068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [term 1 LEADER]: Becoming Leader. State: Replica: 5cda928d8b984c479353e42d12eea535, State: Running, Role: LEADER
I20260809 01:53:24.284936 18068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cda928d8b984c479353e42d12eea535" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 32935 } }
I20260809 01:53:24.285454 18063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:53:24.286646 18069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cda928d8b984c479353e42d12eea535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cda928d8b984c479353e42d12eea535" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 32935 } } }
I20260809 01:53:24.286785 18069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:24.286983 18070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cda928d8b984c479353e42d12eea535. Latest consensus state: current_term: 1 leader_uuid: "5cda928d8b984c479353e42d12eea535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cda928d8b984c479353e42d12eea535" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 32935 } } }
I20260809 01:53:24.287143 18070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535 [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:24.287202 18078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:53:24.290345 18078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:53:24.295830 18078 catalog_manager.cc:1347] Generated new cluster ID: af23fd19aa7b47e09875ee2df2365256
I20260809 01:53:24.295922 18078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:53:24.307165 18078 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:53:24.307529 18078 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:53:24.312561 18078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cda928d8b984c479353e42d12eea535: Generated new TSK 0
I20260809 01:53:24.313194 18078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:53:24.317188 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:32935
--builtin_ntp_servers=127.3.220.20:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 01:53:24.443074 18095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:24.443336 18095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:24.443415 18095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:24.447412 18095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:24.447467 18095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:24.447525 18095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:53:24.452430 18095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:24.453639 18095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:24.455061 18095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:24.464715 18108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:24.464721 18105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:24.464773 18095 server_base.cc:1034] running on GCE node
W20260809 01:53:24.464720 18110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:24.465487 18095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:24.466101 18095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:24.467276 18095 hybrid_clock.cc:648] HybridClock initialized: now 1786240404467263 us; error 35 us; skew 500 ppm
I20260809 01:53:24.468973 18095 webserver.cc:460] Webserver started at http://127.3.220.1:36629/ using document root <none> and password file <none>
I20260809 01:53:24.469636 18095 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:24.469713 18095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:24.469885 18095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:24.471861 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "e5b6cf5557d0449391169f4af79891fe"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "a553aac5d9e2560702e30ac72c280dc4"
server_key_iv: "ca72b72e9990b3fbea57266d6dc6cd77"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.472393 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "e5b6cf5557d0449391169f4af79891fe"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "a553aac5d9e2560702e30ac72c280dc4"
server_key_iv: "ca72b72e9990b3fbea57266d6dc6cd77"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.475966 18095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 01:53:24.478490 18118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.479686 18095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:53:24.479812 18095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "e5b6cf5557d0449391169f4af79891fe"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "a553aac5d9e2560702e30ac72c280dc4"
server_key_iv: "ca72b72e9990b3fbea57266d6dc6cd77"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.479919 18095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:24.484498 18095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:24.485168 18095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:24.485349 18095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:24.485998 18095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:24.486969 18095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:24.487031 18095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.487095 18095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:24.487126 18095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.495718 18095 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:41475
I20260809 01:53:24.495738 18293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:41475 every 8 connection(s)
I20260809 01:53:24.496619 18095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:53:24.502586 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:24.498492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:45869 (local address 127.3.220.62:32935)
0809 01:53:24.498634 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:24.498638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:24.499664 (+ 1026us) server_negotiation.cc:407] Connection header received
0809 01:53:24.500908 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:24.500911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:24.500951 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:24.501030 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:24.502192 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:24.502195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:24.502197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:53:24.502208 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:24.502227 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:53:24.502236 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:24.502311 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:24.502445 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 01:53:24.502482 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:53:24.503394 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:24.498873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:32935 (local address 127.3.220.1:45869)
0809 01:53:24.499501 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:24.499540 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:24.500692 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:24.501187 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:24.501193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:24.501278 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:24.501934 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:53:24.501935 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:24.501977 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:53:24.502043 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:24.502319 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:24.502326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:24.502330 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:53:24.502457 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 01:53:24.502668 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260809 01:53:24.504647 18294 heartbeater.cc:347] Connected to a master server at 127.3.220.62:32935
I20260809 01:53:24.504856 18294 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:24.505221 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 18095
I20260809 01:53:24.505266 18294 heartbeater.cc:511] Master 127.3.220.62:32935 requested a full tablet report, sending...
I20260809 01:53:24.505337 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:53:24.505721 3952 external_mini_cluster.cc:1428] Setting key 8f7980eff3c87c2d28c920ed060227ee
I20260809 01:53:24.507194 18018 ts_manager.cc:194] Registered new tserver with Master: e5b6cf5557d0449391169f4af79891fe (127.3.220.1:41475)
I20260809 01:53:24.507656 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:32935
--builtin_ntp_servers=127.3.220.20:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 01:53:24.621965 18298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:24.622247 18298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:24.622318 18298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:24.626392 18298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:24.626478 18298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:24.626569 18298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:53:24.631295 18298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:24.632422 18298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:24.633915 18298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:24.643877 18311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:24.644049 18298 server_base.cc:1034] running on GCE node
W20260809 01:53:24.643841 18316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:24.643844 18307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:24.644730 18298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:24.645397 18298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:24.646582 18298 hybrid_clock.cc:648] HybridClock initialized: now 1786240404646563 us; error 37 us; skew 500 ppm
I20260809 01:53:24.648530 18298 webserver.cc:460] Webserver started at http://127.3.220.2:33955/ using document root <none> and password file <none>
I20260809 01:53:24.649140 18298 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:24.649235 18298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:24.649495 18298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:24.651690 18298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "488399909fb642ee9dc3097692fc4fa6"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "32f4464042eba45ad7bdbe0a958579e1"
server_key_iv: "eb004f052e5b3a034015c24acf009604"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.652253 18298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "488399909fb642ee9dc3097692fc4fa6"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "32f4464042eba45ad7bdbe0a958579e1"
server_key_iv: "eb004f052e5b3a034015c24acf009604"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.656378 18298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 01:53:24.659050 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.660317 18298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:53:24.660466 18298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "488399909fb642ee9dc3097692fc4fa6"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "32f4464042eba45ad7bdbe0a958579e1"
server_key_iv: "eb004f052e5b3a034015c24acf009604"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.660585 18298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:24.665300 18298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:24.665977 18298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:24.666173 18298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:24.666714 18298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:24.667632 18298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:24.667704 18298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.667773 18298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:24.667822 18298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.676712 18298 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:42263
I20260809 01:53:24.676748 18494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:42263 every 8 connection(s)
I20260809 01:53:24.677735 18298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:53:24.683866 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:24.679628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:50419 (local address 127.3.220.62:32935)
0809 01:53:24.679782 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:24.679786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:24.680866 (+ 1080us) server_negotiation.cc:407] Connection header received
0809 01:53:24.682280 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:24.682283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:24.682323 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:24.682403 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:24.683444 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:24.683447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:24.683449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:53:24.683461 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:24.683482 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:53:24.683491 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:24.683568 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:24.683661 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 01:53:24.683700 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 01:53:24.684754 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:24.680032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:32935 (local address 127.3.220.2:50419)
0809 01:53:24.680713 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:24.680754 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:24.682079 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:24.682561 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:24.682568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:24.682658 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:24.683192 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:53:24.683194 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:24.683232 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:53:24.683299 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:24.683550 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:24.683555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:24.683558 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 01:53:24.683709 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 01:53:24.684006 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260809 01:53:24.685062 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 18298
I20260809 01:53:24.685346 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:53:24.685727 3952 external_mini_cluster.cc:1428] Setting key 18de6c6a68c18e70fd979420bfaf53cb
I20260809 01:53:24.686028 18495 heartbeater.cc:347] Connected to a master server at 127.3.220.62:32935
I20260809 01:53:24.686277 18495 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:24.686810 18495 heartbeater.cc:511] Master 127.3.220.62:32935 requested a full tablet report, sending...
I20260809 01:53:24.687922 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:32935
--builtin_ntp_servers=127.3.220.20:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 01:53:24.687934 18018 ts_manager.cc:194] Registered new tserver with Master: 488399909fb642ee9dc3097692fc4fa6 (127.3.220.2:42263)
W20260809 01:53:24.801190 18498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:24.801445 18498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:24.801483 18498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:24.805505 18498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:24.805579 18498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:24.805635 18498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:53:24.810267 18498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:24.811331 18498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:24.812907 18498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:24.822558 18508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:24.822543 18507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:24.822540 18511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:24.822925 18498 server_base.cc:1034] running on GCE node
I20260809 01:53:24.823318 18498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:24.823948 18498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:24.825137 18498 hybrid_clock.cc:648] HybridClock initialized: now 1786240404825114 us; error 48 us; skew 500 ppm
I20260809 01:53:24.826936 18498 webserver.cc:460] Webserver started at http://127.3.220.3:33405/ using document root <none> and password file <none>
I20260809 01:53:24.827512 18498 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:24.827606 18498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:24.827852 18498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:24.830041 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "f3f8e1871c594d4dbebbd1c1c39a5a2e"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "6354787c907668993080d5c90b13e5af"
server_key_iv: "2c76a57b2a60201dc9c486639607ccd6"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.830607 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "f3f8e1871c594d4dbebbd1c1c39a5a2e"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "6354787c907668993080d5c90b13e5af"
server_key_iv: "2c76a57b2a60201dc9c486639607ccd6"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.834518 18498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:53:24.837157 18521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.838435 18498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:53:24.838572 18498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "f3f8e1871c594d4dbebbd1c1c39a5a2e"
format_stamp: "Formatted at 2026-08-09 01:53:24 on dist-test-slave-gkw7"
server_key: "6354787c907668993080d5c90b13e5af"
server_key_iv: "2c76a57b2a60201dc9c486639607ccd6"
server_key_version: "encryptionkey@0"
I20260809 01:53:24.838688 18498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:24.843595 18498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:24.844208 18498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:24.844394 18498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:24.844995 18498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:24.845963 18498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:24.846033 18498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.846098 18498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:24.846148 18498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:24.855521 18498 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:38611
I20260809 01:53:24.855543 18684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:38611 every 8 connection(s)
I20260809 01:53:24.856485 18498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:53:24.863065 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:24.858455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:60977 (local address 127.3.220.62:32935)
0809 01:53:24.858595 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:24.858599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:24.859758 (+ 1159us) server_negotiation.cc:407] Connection header received
0809 01:53:24.861125 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:24.861129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:24.861173 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:24.861258 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:24.862473 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:24.862477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:24.862480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:53:24.862502 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:24.862525 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:53:24.862534 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:24.862692 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:24.862860 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 01:53:24.862914 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:53:24.864130 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:24.858887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:32935 (local address 127.3.220.3:60977)
0809 01:53:24.859605 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:24.859649 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:24.860909 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:24.861420 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:24.861427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:24.861521 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:24.862153 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:53:24.862155 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:24.862201 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:53:24.862274 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:24.862675 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:24.862685 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:24.862699 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 01:53:24.862909 (+ 210us) client_negotiation.cc:240] Negotiation successful
0809 01:53:24.863185 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":122,"threads_started":1}
I20260809 01:53:24.865244 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 18498
I20260809 01:53:24.865367 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:53:24.865396 18685 heartbeater.cc:347] Connected to a master server at 127.3.220.62:32935
I20260809 01:53:24.865661 18685 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:24.865723 3952 external_mini_cluster.cc:1428] Setting key 497e5256ba5c42b31aaaffe32139cf85
I20260809 01:53:24.866029 18685 heartbeater.cc:511] Master 127.3.220.62:32935 requested a full tablet report, sending...
I20260809 01:53:24.867254 18018 ts_manager.cc:194] Registered new tserver with Master: f3f8e1871c594d4dbebbd1c1c39a5a2e (127.3.220.3:38611)
I20260809 01:53:24.868022 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 01:53:24.871347 18066 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:53:24.870349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.3.220.62:32935)
0809 01:53:24.870511 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:24.870516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:24.870693 (+ 177us) server_negotiation.cc:407] Connection header received
0809 01:53:24.870842 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:24.870845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:24.870880 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:24.870974 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:24.871209 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33430: BlockingRecv error: recv got EOF from 127.0.0.1:33430 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 01:53:24.871804 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 18095
I20260809 01:53:24.888283 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 18298
I20260809 01:53:24.902757 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 18498
I20260809 01:53:24.920166 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 17970
2026-08-09T01:53:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (860 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:53:24 dist-test-slave-gkw7 krb5kdc[18700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:53:24 dist-test-slave-gkw7 krb5kdc[18700](info): set up 2 sockets
Aug 09 01:53:24 dist-test-slave-gkw7 krb5kdc[18700](info): commencing operation
krb5kdc: starting...
W20260809 01:53:26.157466 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:53:26 dist-test-slave-gkw7 krb5kdc[18700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:53:26.289738 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:46477
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:26.404374 18718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:26.404697 18718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:26.404806 18718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:26.408836 18718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:53:26.408931 18718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:26.408969 18718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:26.409027 18718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:53:26.409085 18718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:53:26.414620 18718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:46477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:26.415773 18718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:26.416824 18718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:26.425793 18723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:26.425771 18726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:26.425771 18724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:26.425832 18718 server_base.cc:1034] running on GCE node
I20260809 01:53:26.426636 18718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:26.427758 18718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:26.428966 18718 hybrid_clock.cc:648] HybridClock initialized: now 1786240406428942 us; error 37 us; skew 500 ppm
Aug 09 01:53:26 dist-test-slave-gkw7 krb5kdc[18700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:26.431831 18718 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:53:26.432775 18718 webserver.cc:460] Webserver started at http://127.3.220.62:39595/ using document root <none> and password file <none>
I20260809 01:53:26.433305 18718 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:26.433364 18718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:26.433586 18718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:26.435623 18718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "ac28f48d77f143d78d34af893d895a7e"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "6ce1e91051a1de5ac82608f6d752700c"
server_key_iv: "a46e536ebf9071b16da08f68f6da378c"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.436156 18718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "ac28f48d77f143d78d34af893d895a7e"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "6ce1e91051a1de5ac82608f6d752700c"
server_key_iv: "a46e536ebf9071b16da08f68f6da378c"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.440034 18718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:53:26.442553 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:26.443897 18718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:53:26.444031 18718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "ac28f48d77f143d78d34af893d895a7e"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "6ce1e91051a1de5ac82608f6d752700c"
server_key_iv: "a46e536ebf9071b16da08f68f6da378c"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.444151 18718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:26.457041 18718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:26.460001 18718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:26.460220 18718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:26.467608 18718 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:46477
I20260809 01:53:26.467908 18816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:46477 every 8 connection(s)
I20260809 01:53:26.468604 18718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:53:26.471421 18817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:26.476853 18817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e: Bootstrap starting.
I20260809 01:53:26.477228 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 18718
I20260809 01:53:26.477355 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:53:26.477694 3952 external_mini_cluster.cc:1428] Setting key 46cbc33a7b8bf470e20c22dcfd785a26
I20260809 01:53:26.479681 18817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:26.480423 18817 log.cc:824] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:26.482647 18817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e: No bootstrap required, opened a new log
Aug 09 01:53:26 dist-test-slave-gkw7 krb5kdc[18700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:53:26.487331 18817 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac28f48d77f143d78d34af893d895a7e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46477 } }
I20260809 01:53:26.487576 18817 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:26.487645 18817 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac28f48d77f143d78d34af893d895a7e, State: Initialized, Role: FOLLOWER
I20260809 01:53:26.488099 18817 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac28f48d77f143d78d34af893d895a7e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46477 } }
I20260809 01:53:26.488221 18817 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:53:26.488312 18817 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:53:26.488435 18817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:26.489420 18817 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac28f48d77f143d78d34af893d895a7e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46477 } }
I20260809 01:53:26.489822 18817 leader_election.cc:302] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac28f48d77f143d78d34af893d895a7e; no voters:
I20260809 01:53:26.490144 18817 leader_election.cc:288] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:53:26.490244 18824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:26.490465 18824 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [term 1 LEADER]: Becoming Leader. State: Replica: ac28f48d77f143d78d34af893d895a7e, State: Running, Role: LEADER
I20260809 01:53:26.490818 18824 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac28f48d77f143d78d34af893d895a7e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46477 } }
I20260809 01:53:26.491597 18817 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:53:26.491961 18824 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac28f48d77f143d78d34af893d895a7e. Latest consensus state: current_term: 1 leader_uuid: "ac28f48d77f143d78d34af893d895a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac28f48d77f143d78d34af893d895a7e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46477 } } }
I20260809 01:53:26.492818 18824 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:26.492827 18825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac28f48d77f143d78d34af893d895a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac28f48d77f143d78d34af893d895a7e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 46477 } } }
I20260809 01:53:26.493064 18825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e [sys.catalog]: This master's current role is: LEADER
I20260809 01:53:26.493245 18833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:53:26.494414 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:26.479072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.3.220.62:46477)
0809 01:53:26.479536 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:26.479552 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:26.479586 (+ 34us) server_negotiation.cc:407] Connection header received
0809 01:53:26.480261 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:26.480280 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:26.480658 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:26.481024 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:26.481893 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:26.482951 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:26.484078 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:26.484525 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:26.484656 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:26.487683 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:26.487690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:26.487727 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:26.487765 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:26.489976 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:26.490545 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:26.490551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:26.490558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:26.490698 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:26.491081 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:26.491085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:26.491087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:26.491607 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:26.491843 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:26.492914 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0809 01:53:26.493265 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":146,"threads_started":1}
I20260809 01:53:26.496397 18833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:53:26.501916 18833 catalog_manager.cc:1347] Generated new cluster ID: 66011a7fbb6649a3aa571b5db437c309
I20260809 01:53:26.501986 18833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:53:26.521274 18833 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:53:26.522651 18833 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:53:26.536796 18833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e: Generated new TSK 0
I20260809 01:53:26.537771 18833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:53:26.580269 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46477
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:26.700322 18856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:26.700596 18856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:26.700641 18856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:26.704761 18856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:26.704828 18856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:26.704890 18856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:53:26.710322 18856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:26.711488 18856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:26.712589 18856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:26.722949 18864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:26.722965 18856 server_base.cc:1034] running on GCE node
W20260809 01:53:26.722978 18862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:26.723011 18869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:26.723580 18856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:26.724186 18856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:26.725358 18856 hybrid_clock.cc:648] HybridClock initialized: now 1786240406725336 us; error 32 us; skew 500 ppm
Aug 09 01:53:26 dist-test-slave-gkw7 krb5kdc[18700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:26.728353 18856 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:53:26.729276 18856 webserver.cc:460] Webserver started at http://127.3.220.1:37005/ using document root <none> and password file <none>
I20260809 01:53:26.729895 18856 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:26.729962 18856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:26.730139 18856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:26.732220 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "85a7c3f571694aac8dea95d52cfd90a6"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "1663ff10a3d2359eabe34bff344be006"
server_key_iv: "c6ce1d3f8dacc36587d99d687157a92f"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.732786 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "85a7c3f571694aac8dea95d52cfd90a6"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "1663ff10a3d2359eabe34bff344be006"
server_key_iv: "c6ce1d3f8dacc36587d99d687157a92f"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.736670 18856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260809 01:53:26.739202 18882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:26.740559 18856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:53:26.740724 18856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "85a7c3f571694aac8dea95d52cfd90a6"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "1663ff10a3d2359eabe34bff344be006"
server_key_iv: "c6ce1d3f8dacc36587d99d687157a92f"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.740850 18856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:26.756987 18856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:26.760538 18856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:26.760759 18856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:26.761385 18856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:26.762295 18856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:26.762343 18856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:26.762413 18856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:26.762445 18856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:26.771225 18856 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:46513
I20260809 01:53:26.771293 19064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:46513 every 8 connection(s)
I20260809 01:53:26.772199 18856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:53:26.778286 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 18856
I20260809 01:53:26.778432 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:53:26.778738 3952 external_mini_cluster.cc:1428] Setting key 3c49d53a89f81fb481c961d51e61ca2c
Aug 09 01:53:26 dist-test-slave-gkw7 krb5kdc[18700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:53:26.786929 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:26.774153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:50483 (local address 127.3.220.62:46477)
0809 01:53:26.774301 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:26.774305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:26.775109 (+ 804us) server_negotiation.cc:407] Connection header received
0809 01:53:26.776491 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:26.776495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:26.776556 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:26.776664 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:26.777922 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:26.778525 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:26.779861 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:26.780183 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:26.780291 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:26.783049 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:26.783056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:26.783061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:26.783097 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:26.784620 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:26.785312 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:26.785315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:26.785316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:26.785391 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:26.785780 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:26.785795 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:26.785796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:26.785984 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:26.786097 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:26.786618 (+ 521us) server_negotiation.cc:299] Negotiation successful
0809 01:53:26.786741 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 01:53:26.787738 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:26.774488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:46477 (local address 127.3.220.1:50483)
0809 01:53:26.774955 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:26.774999 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:26.776262 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:26.776851 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:26.776858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:26.777213 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:26.777768 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:26.777784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:26.778668 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:26.778672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:26.779728 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:26.779739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:26.780338 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:26.780343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:26.780514 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:26.781126 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:26.781144 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:26.782872 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:26.784745 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:26.784754 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:26.784773 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:26.785209 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:26.785497 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:26.785500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:26.785502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:26.785678 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:26.786096 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:26.786104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:26.786373 (+ 269us) client_negotiation.cc:769] Sending connection context
0809 01:53:26.786560 (+ 187us) client_negotiation.cc:240] Negotiation successful
0809 01:53:26.786890 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260809 01:53:26.789090 19065 heartbeater.cc:347] Connected to a master server at 127.3.220.62:46477
I20260809 01:53:26.789371 19065 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:53:26.790109 19065 heartbeater.cc:511] Master 127.3.220.62:46477 requested a full tablet report, sending...
I20260809 01:53:26.791692 18759 ts_manager.cc:194] Registered new tserver with Master: 85a7c3f571694aac8dea95d52cfd90a6 (127.3.220.1:46513)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:53:26.793206 18759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:50483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:53:26.808876 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46477
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:26.923714 19073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:26.923977 19073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:26.924055 19073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:26.928061 19073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:26.928115 19073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:26.928216 19073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:53:26.933517 19073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:26.934695 19073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:26.935763 19073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:26.946884 19086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:26.946907 19082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:26.946884 19081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:26.947145 19073 server_base.cc:1034] running on GCE node
I20260809 01:53:26.947571 19073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:26.948274 19073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:26.949488 19073 hybrid_clock.cc:648] HybridClock initialized: now 1786240406949447 us; error 51 us; skew 500 ppm
Aug 09 01:53:26 dist-test-slave-gkw7 krb5kdc[18700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:26.952275 19073 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:53:26.953181 19073 webserver.cc:460] Webserver started at http://127.3.220.2:37857/ using document root <none> and password file <none>
I20260809 01:53:26.953826 19073 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:26.953923 19073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:26.954159 19073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:26.956193 19073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "421cd4b1488d4883b3bf2783689543d1"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "06e08637263f1c5057bba237d98e8409"
server_key_iv: "f2dbd8a25b3738ffb7be6850f8d3f849"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.956773 19073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "421cd4b1488d4883b3bf2783689543d1"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "06e08637263f1c5057bba237d98e8409"
server_key_iv: "f2dbd8a25b3738ffb7be6850f8d3f849"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.960578 19073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:53:26.963224 19094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:26.964480 19073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 01:53:26.964619 19073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "421cd4b1488d4883b3bf2783689543d1"
format_stamp: "Formatted at 2026-08-09 01:53:26 on dist-test-slave-gkw7"
server_key: "06e08637263f1c5057bba237d98e8409"
server_key_iv: "f2dbd8a25b3738ffb7be6850f8d3f849"
server_key_version: "encryptionkey@0"
I20260809 01:53:26.964736 19073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:26.995623 19073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:26.999135 19073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:26.999354 19073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:26.999960 19073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:27.000877 19073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:27.000949 19073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:27.001027 19073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:27.001070 19073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:27.010375 19073 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:38131
I20260809 01:53:27.010399 19256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:38131 every 8 connection(s)
I20260809 01:53:27.011338 19073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:53:27.016698 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 19073
I20260809 01:53:27.016804 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:53:27.017129 3952 external_mini_cluster.cc:1428] Setting key 2ccaac1d0c15367a7d91881df3a4ae23
Aug 09 01:53:27 dist-test-slave-gkw7 krb5kdc[18700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:53:27.026477 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.013169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:35293 (local address 127.3.220.62:46477)
0809 01:53:27.013326 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.013331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.014346 (+ 1015us) server_negotiation.cc:407] Connection header received
0809 01:53:27.015666 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.015669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.015715 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.015794 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:27.017133 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.017851 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.019170 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.019563 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.019690 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.022376 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:27.022379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:27.022381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:27.022404 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:27.024108 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:27.024779 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:27.024782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:27.024784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:27.024860 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:27.025279 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:27.025286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:27.025287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:27.025456 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:27.025576 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.026238 (+ 662us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.026355 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:53:27.027746 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.013669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:46477 (local address 127.3.220.2:35293)
0809 01:53:27.014181 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:27.014233 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:27.015470 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:27.015969 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:27.015976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:27.016397 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:27.016979 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.016996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.018021 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.018024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.019042 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.019053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.019723 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.019725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.019870 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.020515 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:27.020531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:27.022203 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:27.024237 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:27.024246 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:27.024266 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:27.024672 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:27.024960 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:27.024966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:27.024971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:27.025181 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:27.025612 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:27.025622 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:27.025970 (+ 348us) client_negotiation.cc:769] Sending connection context
0809 01:53:27.026209 (+ 239us) client_negotiation.cc:240] Negotiation successful
0809 01:53:27.026594 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:53:27.028990 19257 heartbeater.cc:347] Connected to a master server at 127.3.220.62:46477
I20260809 01:53:27.029309 19257 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:27.029934 19257 heartbeater.cc:511] Master 127.3.220.62:46477 requested a full tablet report, sending...
I20260809 01:53:27.031011 18759 ts_manager.cc:194] Registered new tserver with Master: 421cd4b1488d4883b3bf2783689543d1 (127.3.220.2:38131)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:53:27.031642 18759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:35293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:53:27.048708 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46477
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:53:27.164943 19264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:53:27.165228 19264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:53:27.165395 19264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:53:27.169737 19264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:53:27.169793 19264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:53:27.169886 19264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:53:27.174952 19264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:46477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:53:27.175988 19264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:53:27.177021 19264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:53:27.186810 19274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:27.186882 19270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:53:27.186829 19272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:53:27.187207 19264 server_base.cc:1034] running on GCE node
I20260809 01:53:27.187599 19264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:53:27.188261 19264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:53:27.189451 19264 hybrid_clock.cc:648] HybridClock initialized: now 1786240407189444 us; error 33 us; skew 500 ppm
Aug 09 01:53:27 dist-test-slave-gkw7 krb5kdc[18700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:53:27.192489 19264 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:53:27.193528 19264 webserver.cc:460] Webserver started at http://127.3.220.3:45665/ using document root <none> and password file <none>
I20260809 01:53:27.194295 19264 fs_manager.cc:359] Metadata directory not provided
I20260809 01:53:27.194368 19264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:53:27.194630 19264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:53:27.196949 19264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "606a6af2978d4557949e7ef46f572fa2"
format_stamp: "Formatted at 2026-08-09 01:53:27 on dist-test-slave-gkw7"
server_key: "14110b433db1b7703f9b769f6b9dc53c"
server_key_iv: "f996c9c15c1aa3798146a46bf00a6931"
server_key_version: "encryptionkey@0"
I20260809 01:53:27.197729 19264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "606a6af2978d4557949e7ef46f572fa2"
format_stamp: "Formatted at 2026-08-09 01:53:27 on dist-test-slave-gkw7"
server_key: "14110b433db1b7703f9b769f6b9dc53c"
server_key_iv: "f996c9c15c1aa3798146a46bf00a6931"
server_key_version: "encryptionkey@0"
I20260809 01:53:27.201634 19264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:53:27.204205 19283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:27.205477 19264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:53:27.205655 19264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "606a6af2978d4557949e7ef46f572fa2"
format_stamp: "Formatted at 2026-08-09 01:53:27 on dist-test-slave-gkw7"
server_key: "14110b433db1b7703f9b769f6b9dc53c"
server_key_iv: "f996c9c15c1aa3798146a46bf00a6931"
server_key_version: "encryptionkey@0"
I20260809 01:53:27.205793 19264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:53:27.222482 19264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:53:27.225991 19264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:53:27.226215 19264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:53:27.226818 19264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:53:27.227659 19264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:53:27.227732 19264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:27.227804 19264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:53:27.227859 19264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:53:27.236552 19264 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:43945
I20260809 01:53:27.236580 19464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:43945 every 8 connection(s)
I20260809 01:53:27.237455 19264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:53:27.246531 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 19264
I20260809 01:53:27.246641 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:53:27.246991 3952 external_mini_cluster.cc:1428] Setting key 3e3b2169179b9d5a15b15cb541b7ef16
Aug 09 01:53:27 dist-test-slave-gkw7 krb5kdc[18700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:53:27.251613 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.239374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:46387 (local address 127.3.220.62:46477)
0809 01:53:27.239511 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.239515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.240367 (+ 852us) server_negotiation.cc:407] Connection header received
0809 01:53:27.241792 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.241796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.241869 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.241973 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:27.243142 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.243682 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.244901 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.245120 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.245226 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.247942 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:27.247949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:27.247951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:27.247980 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:27.249597 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:27.250096 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:27.250100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:27.250102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:27.250186 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:27.250541 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:27.250544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:27.250546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:27.250726 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:27.250838 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.251334 (+ 496us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.251453 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:53:27.252439 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.239763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:46477 (local address 127.3.220.3:46387)
0809 01:53:27.240214 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:27.240255 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:27.241512 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:27.242147 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:27.242154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:27.242524 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:53:27.242996 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.243009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.243807 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.243811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.244764 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.244773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.245270 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.245274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.245452 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.246093 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:53:27.246108 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:53:27.247735 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:53:27.249706 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:27.249712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:27.249727 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:27.249989 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:27.250279 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:53:27.250290 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:53:27.250292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:53:27.250435 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:53:27.250850 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:53:27.250857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:53:27.251096 (+ 239us) client_negotiation.cc:769] Sending connection context
0809 01:53:27.251317 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 01:53:27.251656 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260809 01:53:27.253484 19465 heartbeater.cc:347] Connected to a master server at 127.3.220.62:46477
I20260809 01:53:27.253744 19465 heartbeater.cc:464] Registering TS with master...
I20260809 01:53:27.254176 19465 heartbeater.cc:511] Master 127.3.220.62:46477 requested a full tablet report, sending...
I20260809 01:53:27.255318 18759 ts_manager.cc:194] Registered new tserver with Master: 606a6af2978d4557949e7ef46f572fa2 (127.3.220.3:43945)
I20260809 01:53:27.256001 18759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:46387
I20260809 01:53:27.261055 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:53:27.270227 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.263069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.3.220.62:46477)
0809 01:53:27.263169 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.263173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.263392 (+ 219us) server_negotiation.cc:407] Connection header received
0809 01:53:27.263476 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.263478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.263514 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.263581 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:53:27.264272 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.264760 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.265606 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.265811 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.265888 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.267149 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:53:27.267152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:53:27.267154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:53:27.267176 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:53:27.268698 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:27.269161 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:27.269164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:27.269165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:27.269248 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:53:27.269472 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:53:27.269473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:53:27.269474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:53:27.269736 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:53:27.269841 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.270018 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.270107 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 01:53:27.273331 18759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:53:27.278170 18759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:53:27.293412 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.287378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.3.220.3:43945)
0809 01:53:27.287997 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.288009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.288032 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:53:27.288340 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.288348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.288718 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.288916 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:27.289066 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.289896 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.291914 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.292664 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.292987 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.293114 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.293156 (+ 42us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.293225 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":115,"threads_started":1}
I20260809 01:53:27.293406 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.286624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:43945 (local address 127.0.0.1:34514)
0809 01:53:27.287497 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:27.287514 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:27.287607 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:27.288888 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:27.288891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:27.288909 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:27.288986 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.288991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.290133 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.290138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.291784 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.291795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.292823 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.292826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.292961 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.292982 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 01:53:27.293077 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 01:53:27.293252 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":215,"thread_start_us":97,"threads_started":1}
I20260809 01:53:27.293416 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.286928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46513 (local address 127.0.0.1:39012)
0809 01:53:27.287793 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:27.287807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:27.287895 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:27.288578 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:27.288583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:27.288602 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:27.288744 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.288750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.290262 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.290266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.291906 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.291913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.292766 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.292771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.292877 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.292971 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 01:53:27.293079 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 01:53:27.293289 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":429,"thread_start_us":83,"threads_started":1}
I20260809 01:53:27.294060 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.287375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.3.220.1:46513)
0809 01:53:27.287788 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.287804 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.287870 (+ 66us) server_negotiation.cc:407] Connection header received
0809 01:53:27.288010 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.288015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.288242 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.288396 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:27.288901 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.290096 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.291993 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.292607 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.292734 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.292890 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.293281 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.293374 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":114,"threads_started":1}
I20260809 01:53:27.294286 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.286756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.3.220.2:38131)
0809 01:53:27.288130 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.288142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.288162 (+ 20us) server_negotiation.cc:407] Connection header received
0809 01:53:27.288232 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.288236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.288402 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.288542 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:27.288854 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.290097 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.291734 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.292629 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.292842 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.293035 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.294047 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.294144 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":162,"threads_started":1}
I20260809 01:53:27.294605 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.286596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38131 (local address 127.0.0.1:55584)
0809 01:53:27.287394 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:27.287446 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:27.287580 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:27.288545 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:27.288553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:27.288583 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:27.288720 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.288727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.290223 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.290226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.291577 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.291587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.292823 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.292826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.293693 (+ 867us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.293719 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 01:53:27.293797 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 01:53:27.294474 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":866,"thread_start_us":75,"threads_started":1}
I20260809 01:53:27.297163 19355 tablet_service.cc:1467] Processing CreateTablet for tablet ccb17a59038542958975f1f669d31b01 (DEFAULT_TABLE table=test-table [id=84adbeb67b2247f3ae38e5b4d3dc883f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:27.297231 19164 tablet_service.cc:1467] Processing CreateTablet for tablet ccb17a59038542958975f1f669d31b01 (DEFAULT_TABLE table=test-table [id=84adbeb67b2247f3ae38e5b4d3dc883f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:27.297231 18961 tablet_service.cc:1467] Processing CreateTablet for tablet ccb17a59038542958975f1f669d31b01 (DEFAULT_TABLE table=test-table [id=84adbeb67b2247f3ae38e5b4d3dc883f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:53:27.298292 18961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb17a59038542958975f1f669d31b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:27.298302 19164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb17a59038542958975f1f669d31b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:27.298300 19355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb17a59038542958975f1f669d31b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:53:27.304457 19488 tablet_bootstrap.cc:492] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: Bootstrap starting.
I20260809 01:53:27.305915 19492 tablet_bootstrap.cc:492] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: Bootstrap starting.
I20260809 01:53:27.306775 19488 tablet_bootstrap.cc:654] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:27.307981 19488 log.cc:824] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:27.308092 19492 tablet_bootstrap.cc:654] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:27.308913 19492 log.cc:824] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:27.308971 19491 tablet_bootstrap.cc:492] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6: Bootstrap starting.
I20260809 01:53:27.310843 19492 tablet_bootstrap.cc:492] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: No bootstrap required, opened a new log
I20260809 01:53:27.310843 19488 tablet_bootstrap.cc:492] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: No bootstrap required, opened a new log
I20260809 01:53:27.311051 19492 ts_tablet_manager.cc:1397] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:53:27.311051 19488 ts_tablet_manager.cc:1397] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 01:53:27.311066 19491 tablet_bootstrap.cc:654] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6: Neither blocks nor log segments found. Creating new log.
I20260809 01:53:27.311694 19491 log.cc:824] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6: Log is configured to *not* fsync() on all Append() calls
I20260809 01:53:27.313377 19491 tablet_bootstrap.cc:492] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6: No bootstrap required, opened a new log
I20260809 01:53:27.313594 19491 ts_tablet_manager.cc:1397] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:53:27.314970 19492 raft_consensus.cc:348] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.314991 19488 raft_consensus.cc:348] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.315244 19488 raft_consensus.cc:374] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:27.315244 19492 raft_consensus.cc:374] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:27.315305 19492 raft_consensus.cc:729] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421cd4b1488d4883b3bf2783689543d1, State: Initialized, Role: FOLLOWER
I20260809 01:53:27.315310 19488 raft_consensus.cc:729] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606a6af2978d4557949e7ef46f572fa2, State: Initialized, Role: FOLLOWER
I20260809 01:53:27.315776 19488 consensus_queue.cc:260] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.315768 19492 consensus_queue.cc:260] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.316809 19488 ts_tablet_manager.cc:1428] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:53:27.316802 19492 ts_tablet_manager.cc:1428] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:53:27.317205 19257 heartbeater.cc:503] Master 127.3.220.62:46477 was elected leader, sending a full tablet report...
I20260809 01:53:27.317700 19465 heartbeater.cc:503] Master 127.3.220.62:46477 was elected leader, sending a full tablet report...
I20260809 01:53:27.317667 19491 raft_consensus.cc:348] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.317920 19491 raft_consensus.cc:374] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:53:27.317977 19491 raft_consensus.cc:729] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a7c3f571694aac8dea95d52cfd90a6, State: Initialized, Role: FOLLOWER
I20260809 01:53:27.318369 19491 consensus_queue.cc:260] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.319077 19065 heartbeater.cc:503] Master 127.3.220.62:46477 was elected leader, sending a full tablet report...
I20260809 01:53:27.320020 19491 ts_tablet_manager.cc:1428] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:53:27.366230 19500 raft_consensus.cc:482] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:53:27.366514 19500 raft_consensus.cc:504] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.367772 19500 leader_election.cc:288] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 421cd4b1488d4883b3bf2783689543d1 (127.3.220.2:38131), 606a6af2978d4557949e7ef46f572fa2 (127.3.220.3:43945)
I20260809 01:53:27.373266 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.368143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:50185 (local address 127.3.220.2:38131)
0809 01:53:27.368289 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.368294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.368707 (+ 413us) server_negotiation.cc:407] Connection header received
0809 01:53:27.368833 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.368837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.368893 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.368974 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:27.369386 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.370138 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.371946 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.372563 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.372676 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.372742 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.373061 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.373132 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:53:27.373274 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.368283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:48663 (local address 127.3.220.3:43945)
0809 01:53:27.368417 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.368420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.368628 (+ 208us) server_negotiation.cc:407] Connection header received
0809 01:53:27.368834 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.368837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.368893 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.368972 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:53:27.369303 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.370130 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.371946 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.372563 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.372676 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.372742 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.373061 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.373132 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 01:53:27.373338 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.368036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:43945 (local address 127.3.220.1:48663)
0809 01:53:27.368523 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:27.368540 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:27.368693 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:27.369009 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:27.369012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:27.369029 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:27.369165 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.369172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.370249 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.370253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.371810 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.371822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.372694 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.372698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.372891 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.372944 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 01:53:27.373026 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 01:53:27.373130 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":232,"thread_start_us":155,"threads_started":1}
I20260809 01:53:27.373556 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.368037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38131 (local address 127.3.220.1:50185)
0809 01:53:27.368621 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:27.368636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:53:27.368715 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:53:27.369103 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:53:27.369105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:53:27.369118 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:53:27.369293 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.369300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.370249 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.370253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.371817 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:53:27.371822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.372694 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.372698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:53:27.372870 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.372950 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 01:53:27.373026 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 01:53:27.373129 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":265,"thread_start_us":130,"threads_started":1}
I20260809 01:53:27.374153 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb17a59038542958975f1f669d31b01" candidate_uuid: "85a7c3f571694aac8dea95d52cfd90a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "421cd4b1488d4883b3bf2783689543d1" is_pre_election: true
I20260809 01:53:27.374152 19387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb17a59038542958975f1f669d31b01" candidate_uuid: "85a7c3f571694aac8dea95d52cfd90a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606a6af2978d4557949e7ef46f572fa2" is_pre_election: true
I20260809 01:53:27.374488 19193 raft_consensus.cc:2454] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a7c3f571694aac8dea95d52cfd90a6 in term 0.
I20260809 01:53:27.374488 19387 raft_consensus.cc:2454] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a7c3f571694aac8dea95d52cfd90a6 in term 0.
I20260809 01:53:27.375013 18886 leader_election.cc:302] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 421cd4b1488d4883b3bf2783689543d1, 606a6af2978d4557949e7ef46f572fa2, 85a7c3f571694aac8dea95d52cfd90a6; no voters:
I20260809 01:53:27.375310 19500 raft_consensus.cc:2781] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:53:27.375422 19500 raft_consensus.cc:482] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:53:27.375488 19500 raft_consensus.cc:3037] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:27.376575 19500 raft_consensus.cc:504] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.376948 19500 leader_election.cc:288] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [CANDIDATE]: Term 1 election: Requested vote from peers 421cd4b1488d4883b3bf2783689543d1 (127.3.220.2:38131), 606a6af2978d4557949e7ef46f572fa2 (127.3.220.3:43945)
I20260809 01:53:27.377398 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb17a59038542958975f1f669d31b01" candidate_uuid: "85a7c3f571694aac8dea95d52cfd90a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "421cd4b1488d4883b3bf2783689543d1"
I20260809 01:53:27.377411 19387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb17a59038542958975f1f669d31b01" candidate_uuid: "85a7c3f571694aac8dea95d52cfd90a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606a6af2978d4557949e7ef46f572fa2"
I20260809 01:53:27.377573 19387 raft_consensus.cc:3037] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:27.377568 19193 raft_consensus.cc:3037] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:53:27.378688 19387 raft_consensus.cc:2454] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a7c3f571694aac8dea95d52cfd90a6 in term 1.
I20260809 01:53:27.378854 19193 raft_consensus.cc:2454] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a7c3f571694aac8dea95d52cfd90a6 in term 1.
I20260809 01:53:27.379036 18886 leader_election.cc:302] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606a6af2978d4557949e7ef46f572fa2, 85a7c3f571694aac8dea95d52cfd90a6; no voters:
I20260809 01:53:27.379225 19500 raft_consensus.cc:2781] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:53:27.379474 19500 raft_consensus.cc:686] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 1 LEADER]: Becoming Leader. State: Replica: 85a7c3f571694aac8dea95d52cfd90a6, State: Running, Role: LEADER
I20260809 01:53:27.379855 19500 consensus_queue.cc:237] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } }
I20260809 01:53:27.383478 18757 catalog_manager.cc:5495] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a7c3f571694aac8dea95d52cfd90a6 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "85a7c3f571694aac8dea95d52cfd90a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85a7c3f571694aac8dea95d52cfd90a6" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46513 } health_report { overall_health: HEALTHY } } }
I20260809 01:53:27.415992 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.412544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.3.220.1:46513)
0809 01:53:27.412681 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 01:53:27.412685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:53:27.412723 (+ 38us) server_negotiation.cc:407] Connection header received
0809 01:53:27.412803 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:53:27.412805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:53:27.412867 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:53:27.412950 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:53:27.413197 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.413739 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.414597 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:53:27.414840 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:53:27.414933 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:53:27.415199 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:53:27.415661 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:53:27.415757 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 01:53:27.415810 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 01:53:27.415864 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:53:27.422282 19193 raft_consensus.cc:1261] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 1 FOLLOWER]: Refusing update from remote peer 85a7c3f571694aac8dea95d52cfd90a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:27.422279 19387 raft_consensus.cc:1261] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 1 FOLLOWER]: Refusing update from remote peer 85a7c3f571694aac8dea95d52cfd90a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:53:27.423122 19504 consensus_queue.cc:1035] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606a6af2978d4557949e7ef46f572fa2" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:27.423377 19500 consensus_queue.cc:1035] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "421cd4b1488d4883b3bf2783689543d1" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:53:27.430948 19508 mvcc.cc:205] Tried to move back new op lower bound from 7316440708793573376 to 7316440708630892544. Current Snapshot: MvccSnapshot[applied={T|T < 7316440708793573376}]
I20260809 01:53:27.431790 19509 mvcc.cc:205] Tried to move back new op lower bound from 7316440708793573376 to 7316440708630892544. Current Snapshot: MvccSnapshot[applied={T|T < 7316440708793573376}]
I20260809 01:53:27.439165 18757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:27.439361 18757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:53:27.441879 18757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac28f48d77f143d78d34af893d895a7e: Sending DeleteTablet for 3 replicas of tablet ccb17a59038542958975f1f669d31b01
I20260809 01:53:27.442682 19164 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb17a59038542958975f1f669d31b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:27 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:55584
I20260809 01:53:27.442960 18961 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb17a59038542958975f1f669d31b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:27 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:39012
I20260809 01:53:27.443223 19355 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb17a59038542958975f1f669d31b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:53:27 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:34514
I20260809 01:53:27.443303 19523 tablet_replica.cc:330] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: stopping tablet replica
I20260809 01:53:27.443588 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 18856
I20260809 01:53:27.443710 19524 tablet_replica.cc:330] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6: stopping tablet replica
I20260809 01:53:27.444021 19524 raft_consensus.cc:2229] T ccb17a59038542958975f1f669d31b01 P 85a7c3f571694aac8dea95d52cfd90a6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:53:27.449424 19523 raft_consensus.cc:2229] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:27.449833 19523 raft_consensus.cc:2258] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:27.452641 19525 tablet_replica.cc:330] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: stopping tablet replica
I20260809 01:53:27.452756 19523 ts_tablet_manager.cc:1905] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:27.452996 19525 raft_consensus.cc:2229] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:53:27.453305 19525 raft_consensus.cc:2258] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:53:27.454555 19525 ts_tablet_manager.cc:1905] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:53:27.456641 19525 ts_tablet_manager.cc:1918] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:53:27.456773 19525 log.cc:1196] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/ccb17a59038542958975f1f669d31b01
I20260809 01:53:27.456811 19523 ts_tablet_manager.cc:1918] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:53:27.456924 19523 log.cc:1196] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/ccb17a59038542958975f1f669d31b01
I20260809 01:53:27.457155 19525 ts_tablet_manager.cc:1939] T ccb17a59038542958975f1f669d31b01 P 606a6af2978d4557949e7ef46f572fa2: Deleting consensus metadata
I20260809 01:53:27.457325 19523 ts_tablet_manager.cc:1939] T ccb17a59038542958975f1f669d31b01 P 421cd4b1488d4883b3bf2783689543d1: Deleting consensus metadata
I20260809 01:53:27.458455 18738 catalog_manager.cc:4841] TS 606a6af2978d4557949e7ef46f572fa2 (127.3.220.3:43945): tablet ccb17a59038542958975f1f669d31b01 (table test-table [id=84adbeb67b2247f3ae38e5b4d3dc883f]) successfully deleted
I20260809 01:53:27.458576 18736 catalog_manager.cc:4841] TS 421cd4b1488d4883b3bf2783689543d1 (127.3.220.2:38131): tablet ccb17a59038542958975f1f669d31b01 (table test-table [id=84adbeb67b2247f3ae38e5b4d3dc883f]) successfully deleted
W20260809 01:53:27.459961 18739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:46513 (error 108)
I20260809 01:53:27.460246 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 19073
I20260809 01:53:27.467223 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:53:27.466868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46513 (local address 127.0.0.1:39042)
0809 01:53:27.467015 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 01:53:27.467108 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:46513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 01:53:27.467489 18739 catalog_manager.cc:4584] TS 85a7c3f571694aac8dea95d52cfd90a6 (127.3.220.1:46513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccb17a59038542958975f1f669d31b01: Network error: Client connection negotiation failed: client connection to 127.3.220.1:46513: connect: Connection refused (error 111)
I20260809 01:53:27.472381 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 19264
I20260809 01:53:27.485206 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 18718
2026-08-09T01:53:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2566 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:53:27 dist-test-slave-gkw7 krb5kdc[19531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:53:27 dist-test-slave-gkw7 krb5kdc[19531](info): set up 2 sockets
Aug 09 01:53:27 dist-test-slave-gkw7 krb5kdc[19531](info): commencing operation
krb5kdc: starting...
W20260809 01:53:28.764560 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:53:28 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:53:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.220.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.220.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/rangeradmin_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/rangeradmin_127.3.220.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.220.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.220.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/rangerlookup_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/rangerlookup_127.3.220.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.220.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.220.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/HTTP_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/HTTP_127.3.220.20@KRBTEST.COM.keytab.
I20260809 01:53:28.950907 3952 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIuGTYa/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/postgres -l logfile start
2026-08-09 01:53:31.482 UTC [19578] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 01:53:31.482 UTC [19578] LOG: listening on IPv4 address "127.3.220.20", port 47321
2026-08-09 01:53:31.486 UTC [19578] LOG: listening on Unix socket "/tmp/.s.PGSQL.47321"
2026-08-09 01:53:31.524 UTC [19581] LOG: database system was shut down at 2026-08-09 01:53:30 UTC
2026-08-09 01:53:31.529 UTC [19578] LOG: database system is ready to accept connections
I20260809 01:53:32.743687 3952 mini_postgres.cc:96] Postgres bound to 47321
2026-08-09 01:53:32.751 UTC [19592] FATAL: database "slave" does not exist
127.3.220.20:47321 - accepting connections
I20260809 01:53:32.752234 3952 mini_ranger.cc:161] Starting Ranger...
I20260809 01:53:32.768468 3952 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 01:53:33.272295 3952 mini_ranger.cc:87] Created ranger Postgres database
I20260809 01:53:33.272406 3952 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-admin
2026-08-09 01:53:33,566 [I] DB FLAVOR :POSTGRES
2026-08-09 01:53:33,566 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 01:53:33,566 [I] Checking connection..
2026-08-09 01:53:33,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 01:53:33,805 [I] Checking connection passed.
2026-08-09 01:53:33,805 [I] --------- Verifying version history table ---------
2026-08-09 01:53:33,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 01:53:34,061 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 01:53:34,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 01:53:34,332 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 01:53:34,333 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 01:53:34,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 01:53:34.596 UTC [19682] WARNING: there is no transaction in progress
2026-08-09 01:53:34,605 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 01:53:34,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 01:53:34,873 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 01:53:34,873 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 01:53:34,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 01:53:35,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 01:53:35,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gkw7.c.gcp-upstream.internal','N') ;"
2026-08-09 01:53:35,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 01:53:35,916 [I] Table x_portal_user does not exist in database ranger
2026-08-09 01:53:35,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 01:53:36,174 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 01:53:36,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 01:53:36,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 01:53:36,713 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 01:53:36,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 01:53:37.718 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.735 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.752 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.768 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.786 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.806 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.861 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.868 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.883 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.894 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.904 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.918 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.927 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.934 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.941 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:37.948 UTC [19852] WARNING: there is no transaction in progress
2026-08-09 01:53:38,619 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 01:53:38,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 01:53:38,860 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 01:53:38,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 01:53:39,102 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 01:53:39,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 01:53:39,377 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 01:53:39,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 01:53:39,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 01:53:39,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gkw7.c.gcp-upstream.internal';"
2026-08-09 01:53:40,134 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 01:53:40,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 01:53:40,395 [I] DB_PATCHES have already been applied
I20260809 01:53:40.401647 3952 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIuGTYa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 01:53:40.401711 3952 mini_ranger.cc:193] Using host: 127.3.220.20
I20260809 01:53:40.406253 3952 mini_ranger.cc:227] Ranger admin URL: http://127.3.220.20:38989
Aug 09, 2026 1:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 1:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 1:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 1:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 1:53:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 1:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 1:53:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 01:53:41.760567 3952 mini_ranger.cc:160] Time spent starting Ranger: real 9.008s user 0.000s sys 0.014s
Aug 09, 2026 1:54:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 1:54:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 1:54:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 1:54:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 1:54:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 01:54:03.247740 3952 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.220.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.220.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/rangerkms_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/rangerkms_127.3.220.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.220.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.220.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.220.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/HTTP_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.220.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/HTTP_127.3.220.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 01:54:03 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240443, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 01:54:03.339560 3952 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 01:54:03.350697 3952 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 01:54:04.279565 3952 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 01:54:04.279690 3952 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms
I20260809 01:54:04.279803 3952 mini_ranger_kms.cc:227] Using postgres at 127.3.220.20:47321
2026-08-09 01:54:04,510 [I] DB FLAVOR :POSTGRES
2026-08-09 01:54:04,511 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 01:54:04,511 [I] Checking connection
2026-08-09 01:54:04,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 01:54:04,753 [I] connection success
2026-08-09 01:54:04,753 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 01:54:04,753 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 01:54:04,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 01:54:05,010 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 01:54:05,010 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 01:54:05,010 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 01:54:05,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:47321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 01:54:05,298 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 01:54:05.800410 3952 mini_ranger_kms.cc:310] Created kms service
I20260809 01:54:05.993963 3952 mini_ranger_kms.cc:326] Created kudu user
I20260809 01:54:06.041703 3952 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 01:54:06 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240443, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:54:06.451663 3952 mini_ranger_kms.cc:384] Added ranger policy
I20260809 01:54:06.451809 3952 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 01:54:06.451831 3952 mini_ranger_kms.cc:240] Using host: 127.3.220.20
I20260809 01:54:06.457648 3952 mini_ranger_kms.cc:276] Ranger KMS PID: 20123
I20260809 01:54:06.457795 3952 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.220.20:58947
Aug 09, 2026 1:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 1:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 1:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 1:54:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 1:54:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 1:54:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 01:54:08 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786240448, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.220.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:54:09.266597 3952 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.927s user 0.024s sys 0.190s
I20260809 01:54:09.266747 3952 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 01:54:09.266866 3952 mini_ranger_kms.cc:401] 127.3.220.20:58947/kms/v1/keys
Aug 09 01:54:09 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240448, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.220.20@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
Aug 09 01:54:09 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240448, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.220.20@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
Aug 09 01:54:12 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:54:12.129066 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44581
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:44581
--ranger_config_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:58947/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:54:12.244452 20197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:54:12.244729 20197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:54:12.244769 20197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:54:12.248929 20197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:54:12.248991 20197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:54:12.249009 20197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:54:12.249027 20197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:54:12.249074 20197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:54:12.254581 20197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.220.20:58947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:44581
--ranger_config_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:54:12.255642 20197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:54:12.256716 20197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:54:12.265900 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:54:12.265924 20207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:54:12.265890 20202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:54:12.266355 20197 server_base.cc:1034] running on GCE node
I20260809 01:54:12.266935 20197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:54:12.267995 20197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:54:12.269219 20197 hybrid_clock.cc:648] HybridClock initialized: now 1786240452269197 us; error 37 us; skew 500 ppm
Aug 09 01:54:12 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:54:12.276103 20197 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:54:12.277043 20197 webserver.cc:460] Webserver started at http://127.3.220.62:35671/ using document root <none> and password file <none>
I20260809 01:54:12.277638 20197 fs_manager.cc:359] Metadata directory not provided
I20260809 01:54:12.277704 20197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:54:12.277887 20197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 01:54:12 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:54:12.336534 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "50c0e627a91f45b3bdf31d85b6888408"
format_stamp: "Formatted at 2026-08-09 01:54:12 on dist-test-slave-gkw7"
server_key: "68a00d1de8f7936014c9f9965b8560ba"
server_key_iv: "b42de6737e5668b1b4a5d1a21a7e8a16"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:12.337399 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "50c0e627a91f45b3bdf31d85b6888408"
format_stamp: "Formatted at 2026-08-09 01:54:12 on dist-test-slave-gkw7"
server_key: "68a00d1de8f7936014c9f9965b8560ba"
server_key_iv: "b42de6737e5668b1b4a5d1a21a7e8a16"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:12.342561 20197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 01:54:12.358258 20217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:12.359910 20197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:54:12.360138 20197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "50c0e627a91f45b3bdf31d85b6888408"
format_stamp: "Formatted at 2026-08-09 01:54:12 on dist-test-slave-gkw7"
server_key: "68a00d1de8f7936014c9f9965b8560ba"
server_key_iv: "b42de6737e5668b1b4a5d1a21a7e8a16"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:12.360306 20197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:54:12.377964 20197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:54:12.381438 20197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:54:12.381711 20197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:54:12.390877 20197 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:44581
I20260809 01:54:12.390933 20297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:44581 every 8 connection(s)
I20260809 01:54:12.391981 20197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:54:12.395241 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:54:12.398509 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 20197
I20260809 01:54:12.398649 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:54:12.398957 3952 external_mini_cluster.cc:1428] Setting key 428a2737c2ddb94a3ee3d3bc71af4a90
I20260809 01:54:12.402391 20298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408: Bootstrap starting.
I20260809 01:54:12.405393 20298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408: Neither blocks nor log segments found. Creating new log.
I20260809 01:54:12.406318 20298 log.cc:824] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408: Log is configured to *not* fsync() on all Append() calls
I20260809 01:54:12.408407 20298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408: No bootstrap required, opened a new log
Aug 09 01:54:12 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:54:12.412923 20298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c0e627a91f45b3bdf31d85b6888408" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44581 } }
I20260809 01:54:12.413151 20298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:54:12.413215 20298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c0e627a91f45b3bdf31d85b6888408, State: Initialized, Role: FOLLOWER
I20260809 01:54:12.413765 20298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c0e627a91f45b3bdf31d85b6888408" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44581 } }
I20260809 01:54:12.413967 20298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:54:12.414053 20298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:54:12.414196 20298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:54:12.414947 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:12.400514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.3.220.62:44581)
0809 01:54:12.401019 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:12.401041 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:12.401088 (+ 47us) server_negotiation.cc:407] Connection header received
0809 01:54:12.402042 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:12.402068 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:12.402525 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:12.402909 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:54:12.403636 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:12.405263 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:12.406259 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:12.406976 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:12.407148 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:12.410135 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:54:12.410141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:54:12.410154 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:54:12.410181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:54:12.411841 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:12.412408 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:12.412412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:12.412417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:12.412529 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:12.412838 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:12.412841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:12.412842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:12.413320 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:54:12.413570 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:12.413861 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 01:54:12.414161 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":135,"threads_started":1}
I20260809 01:54:12.415213 20298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c0e627a91f45b3bdf31d85b6888408" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44581 } }
I20260809 01:54:12.415570 20298 leader_election.cc:302] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50c0e627a91f45b3bdf31d85b6888408; no voters:
I20260809 01:54:12.415967 20298 leader_election.cc:288] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:54:12.416020 20305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:54:12.416254 20305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [term 1 LEADER]: Becoming Leader. State: Replica: 50c0e627a91f45b3bdf31d85b6888408, State: Running, Role: LEADER
I20260809 01:54:12.416574 20305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c0e627a91f45b3bdf31d85b6888408" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44581 } }
I20260809 01:54:12.417343 20298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:54:12.417904 20309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50c0e627a91f45b3bdf31d85b6888408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c0e627a91f45b3bdf31d85b6888408" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44581 } } }
I20260809 01:54:12.418035 20309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [sys.catalog]: This master's current role is: LEADER
I20260809 01:54:12.418337 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c0e627a91f45b3bdf31d85b6888408. Latest consensus state: current_term: 1 leader_uuid: "50c0e627a91f45b3bdf31d85b6888408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c0e627a91f45b3bdf31d85b6888408" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44581 } } }
I20260809 01:54:12.418442 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408 [sys.catalog]: This master's current role is: LEADER
I20260809 01:54:12.419745 20315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:54:12.420025 20298 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 01:54:12.427014 20315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:54:12.434590 20315 catalog_manager.cc:1347] Generated new cluster ID: 47a720f706554d3d9639d3276edf5f6a
I20260809 01:54:12.434675 20315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:54:12.453998 20315 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:54:12.455318 20315 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:54:12.461826 20315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408: Generated new TSK 0
I20260809 01:54:12.462637 20315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:54:13.230476 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:58947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44581
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:54:13.364251 20368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:54:13.364547 20368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:54:13.364624 20368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:54:13.368954 20368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:54:13.369089 20368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:54:13.369217 20368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:54:13.377172 20368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.220.20:58947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:54:13.378611 20368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:54:13.379951 20368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:54:13.391848 20380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:54:13.392204 20376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:54:13.392740 20377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:54:13.393698 20368 server_base.cc:1034] running on GCE node
I20260809 01:54:13.394289 20368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:54:13.395121 20368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:54:13.396368 20368 hybrid_clock.cc:648] HybridClock initialized: now 1786240453396339 us; error 44 us; skew 500 ppm
Aug 09 01:54:13 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:54:13.400146 20368 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:54:13.401242 20368 webserver.cc:460] Webserver started at http://127.3.220.1:40841/ using document root <none> and password file <none>
I20260809 01:54:13.402024 20368 fs_manager.cc:359] Metadata directory not provided
I20260809 01:54:13.402184 20368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:54:13.402477 20368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 01:54:13 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:54:13.421919 20368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "21d9d1f14e2a4fc0b41e92829043f40a"
format_stamp: "Formatted at 2026-08-09 01:54:13 on dist-test-slave-gkw7"
server_key: "620128072cb535e6722af302dd79e4a9"
server_key_iv: "25e5ebc2c8cdab5a625bb2567585ccf7"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:13.422729 20368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "21d9d1f14e2a4fc0b41e92829043f40a"
format_stamp: "Formatted at 2026-08-09 01:54:13 on dist-test-slave-gkw7"
server_key: "620128072cb535e6722af302dd79e4a9"
server_key_iv: "25e5ebc2c8cdab5a625bb2567585ccf7"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:13.443796 20368 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.005s sys 0.000s
I20260809 01:54:13.465204 20397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:13.466971 20368 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260809 01:54:13.467108 20368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "21d9d1f14e2a4fc0b41e92829043f40a"
format_stamp: "Formatted at 2026-08-09 01:54:13 on dist-test-slave-gkw7"
server_key: "620128072cb535e6722af302dd79e4a9"
server_key_iv: "25e5ebc2c8cdab5a625bb2567585ccf7"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:13.467301 20368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:54:13.482581 20368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:54:13.494875 20368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:54:13.495111 20368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:54:13.495895 20368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:54:13.497079 20368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:54:13.497149 20368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:13.497213 20368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:54:13.497248 20368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:13.509719 20368 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:43305
I20260809 01:54:13.511086 20368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:54:13.516239 20579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:43305 every 8 connection(s)
I20260809 01:54:13.520744 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 20368
I20260809 01:54:13.520888 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:54:13.521236 3952 external_mini_cluster.cc:1428] Setting key 482b022d069f1fcc5800d928f753ce83
Aug 09 01:54:13 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:54:13.544982 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:13.518727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:60309 (local address 127.3.220.62:44581)
0809 01:54:13.530395 (+ 11668us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:13.530401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:13.530417 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:54:13.530462 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:13.530465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:13.530514 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:13.530628 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:54:13.532356 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:13.533237 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:13.535008 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:13.535357 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:13.535469 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:13.539786 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:54:13.539790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:54:13.539793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:54:13.539826 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:54:13.541890 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:13.542558 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:13.542564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:13.542566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:13.542683 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:13.543120 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:13.543123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:13.543125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:13.543377 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:54:13.543479 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:13.544621 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0809 01:54:13.544773 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11570,"thread_start_us":103,"threads_started":1}
I20260809 01:54:13.547119 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:13.519314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44581 (local address 127.3.220.1:60309)
0809 01:54:13.526935 (+ 7621us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:13.527012 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 01:54:13.528179 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:54:13.530991 (+ 2812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:54:13.531002 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:54:13.531381 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:54:13.532149 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:13.532175 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:13.533396 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:13.533400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:13.534857 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:13.534869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:13.535686 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:13.535692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:13.535914 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:13.537703 (+ 1789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:54:13.537724 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:54:13.539506 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:54:13.542034 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:54:13.542041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:54:13.542060 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:54:13.542430 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:54:13.542795 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:54:13.542798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:54:13.542800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:54:13.542994 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:54:13.543541 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:54:13.543548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:54:13.543950 (+ 402us) client_negotiation.cc:769] Sending connection context
0809 01:54:13.545441 (+ 1491us) client_negotiation.cc:240] Negotiation successful
0809 01:54:13.545859 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7335,"thread_start_us":161,"threads_started":1}
I20260809 01:54:13.549113 20580 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44581
I20260809 01:54:13.549386 20580 heartbeater.cc:464] Registering TS with master...
I20260809 01:54:13.550046 20580 heartbeater.cc:511] Master 127.3.220.62:44581 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:54:13.552060 20238 ts_manager.cc:194] Registered new tserver with Master: 21d9d1f14e2a4fc0b41e92829043f40a (127.3.220.1:43305)
I20260809 01:54:13.553745 20238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:60309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
Aug 09 01:54:13 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:54:13.596627 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:58947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44581
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:54:13.776986 20590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:54:13.777891 20590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:54:13.778053 20590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:54:13.784365 20590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:54:13.784539 20590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:54:13.784668 20590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:54:13.792786 20590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.220.20:58947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:54:13.794549 20590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:54:13.796082 20590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:54:13.810254 20600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:54:13.810720 20603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:54:13.811754 20590 server_base.cc:1034] running on GCE node
W20260809 01:54:13.812171 20599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:54:13.812741 20590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:54:13.813612 20590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:54:13.845247 20590 hybrid_clock.cc:648] HybridClock initialized: now 1786240453845134 us; error 116 us; skew 500 ppm
Aug 09 01:54:13 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:54:13.850364 20590 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:54:13.851711 20590 webserver.cc:460] Webserver started at http://127.3.220.2:37883/ using document root <none> and password file <none>
I20260809 01:54:13.852552 20590 fs_manager.cc:359] Metadata directory not provided
I20260809 01:54:13.852721 20590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:54:13.853030 20590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 01:54:13 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:54:13.870447 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "5ca2145750f0402684242c3f09dc19fc"
format_stamp: "Formatted at 2026-08-09 01:54:13 on dist-test-slave-gkw7"
server_key: "ee24e1c0c96e6861154ad5d2c032ad0b"
server_key_iv: "c6dcad048cc7704784f507e8cdc97671"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:13.871335 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "5ca2145750f0402684242c3f09dc19fc"
format_stamp: "Formatted at 2026-08-09 01:54:13 on dist-test-slave-gkw7"
server_key: "ee24e1c0c96e6861154ad5d2c032ad0b"
server_key_iv: "c6dcad048cc7704784f507e8cdc97671"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:13.876781 20590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 01:54:13.911450 20611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:13.913296 20590 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.001s
I20260809 01:54:13.913457 20590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "5ca2145750f0402684242c3f09dc19fc"
format_stamp: "Formatted at 2026-08-09 01:54:13 on dist-test-slave-gkw7"
server_key: "ee24e1c0c96e6861154ad5d2c032ad0b"
server_key_iv: "c6dcad048cc7704784f507e8cdc97671"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:13.913638 20590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:54:13.939749 20590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:54:13.943586 20590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:54:13.943790 20590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:54:13.944500 20590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:54:13.945703 20590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:54:13.945768 20590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:13.945822 20590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:54:13.945853 20590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:13.959911 20590 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:41775
I20260809 01:54:13.961078 20590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:54:13.963011 20791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:41775 every 8 connection(s)
I20260809 01:54:13.966117 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 20590
I20260809 01:54:13.966269 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:54:13.966607 3952 external_mini_cluster.cc:1428] Setting key c40ecbeae344424b3f60fff8ea188721
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Aug 09 01:54:13 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:54:14.002115 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:13.965776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:36313 (local address 127.3.220.62:44581)
0809 01:54:13.966457 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:13.966464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:13.978197 (+ 11733us) server_negotiation.cc:407] Connection header received
0809 01:54:13.986655 (+ 8458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:13.986660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:13.986739 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:13.986819 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:54:13.988746 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:13.989653 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:13.991424 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:13.991808 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:13.991893 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:13.997482 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:54:13.997487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:54:13.997490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:54:13.997524 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:54:13.999403 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:14.000092 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:14.000096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:14.000098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:14.000192 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:14.000616 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:14.000620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:14.000622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:14.000863 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:54:14.000936 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.001730 (+ 794us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.001912 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567}
I20260809 01:54:14.004279 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:13.965330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44581 (local address 127.3.220.2:36313)
0809 01:54:13.977965 (+ 12635us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:13.978053 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0809 01:54:13.979297 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:54:13.987363 (+ 8066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:54:13.987375 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:54:13.987808 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:54:13.988548 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:13.988569 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:13.989836 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:13.989839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:13.991264 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:13.991277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:13.992120 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:13.992123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:13.992337 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:13.995375 (+ 3038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:54:13.995399 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:54:13.997256 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:54:13.999563 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:54:13.999569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:54:13.999589 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:54:13.999964 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:54:14.000310 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:54:14.000313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:54:14.000315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:54:14.000506 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:54:14.001026 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:54:14.001033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:54:14.001408 (+ 375us) client_negotiation.cc:769] Sending connection context
0809 01:54:14.002632 (+ 1224us) client_negotiation.cc:240] Negotiation successful
0809 01:54:14.003082 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12320,"thread_start_us":150,"threads_started":1}
I20260809 01:54:14.006618 20792 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44581
I20260809 01:54:14.006896 20792 heartbeater.cc:464] Registering TS with master...
I20260809 01:54:14.007586 20792 heartbeater.cc:511] Master 127.3.220.62:44581 requested a full tablet report, sending...
I20260809 01:54:14.008858 20238 ts_manager.cc:194] Registered new tserver with Master: 5ca2145750f0402684242c3f09dc19fc (127.3.220.2:41775)
I20260809 01:54:14.009800 20238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:36313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:54:14.021420 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:58947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44581
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:54:14 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
W20260809 01:54:14.178767 20797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:54:14.179142 20797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:54:14.179215 20797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:54:14.185041 20797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:54:14.185138 20797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:54:14.185225 20797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:54:14.194679 20797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.220.20:58947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:54:14.196161 20797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:54:14.197886 20797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:54:14.216459 20808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:54:14.217056 20803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:54:14.216468 20804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:54:14.222937 20797 server_base.cc:1034] running on GCE node
I20260809 01:54:14.223520 20797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:54:14.224462 20797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:54:14.225632 20797 hybrid_clock.cc:648] HybridClock initialized: now 1786240454225612 us; error 48 us; skew 500 ppm
Aug 09 01:54:14 dist-test-slave-gkw7 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:54:14.229322 20797 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:54:14.230484 20797 webserver.cc:460] Webserver started at http://127.3.220.3:46431/ using document root <none> and password file <none>
I20260809 01:54:14.231204 20797 fs_manager.cc:359] Metadata directory not provided
I20260809 01:54:14.231290 20797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:54:14.231572 20797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 01:54:14 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:54:14.254681 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "d1743ee3932546c6aeb9feb4f87c7893"
format_stamp: "Formatted at 2026-08-09 01:54:14 on dist-test-slave-gkw7"
server_key: "74f63b3ea565000767fb5988cbeb7455"
server_key_iv: "66fa9fbe7f96dbde34d4285e47f6d4e4"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:14.255856 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "d1743ee3932546c6aeb9feb4f87c7893"
format_stamp: "Formatted at 2026-08-09 01:54:14 on dist-test-slave-gkw7"
server_key: "74f63b3ea565000767fb5988cbeb7455"
server_key_iv: "66fa9fbe7f96dbde34d4285e47f6d4e4"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:14.261497 20797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 01:54:14.279440 20817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:14.281296 20797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:54:14.281453 20797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "d1743ee3932546c6aeb9feb4f87c7893"
format_stamp: "Formatted at 2026-08-09 01:54:14 on dist-test-slave-gkw7"
server_key: "74f63b3ea565000767fb5988cbeb7455"
server_key_iv: "66fa9fbe7f96dbde34d4285e47f6d4e4"
server_key_version: "kuduclusterkey@0"
I20260809 01:54:14.281642 20797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:54:14.313498 20797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:54:14.317483 20797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:54:14.317740 20797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:54:14.318598 20797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:54:14.319867 20797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:54:14.319929 20797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:14.319988 20797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:54:14.320024 20797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:54:14.332607 20797 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:44653
I20260809 01:54:14.332686 20989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:44653 every 8 connection(s)
I20260809 01:54:14.333892 20797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:54:14.334771 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 20797
I20260809 01:54:14.334913 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:54:14.335265 3952 external_mini_cluster.cc:1428] Setting key 5edc11148f4f2a2d4dd173a2e1c15e7f
Aug 09 01:54:14 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:54:14.351187 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.337950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:59827 (local address 127.3.220.62:44581)
0809 01:54:14.338067 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:14.338071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:14.338089 (+ 18us) server_negotiation.cc:407] Connection header received
0809 01:54:14.338834 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:14.338838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:14.338895 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:14.338984 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:54:14.340412 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.341247 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.343027 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.343379 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.343497 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.346554 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:54:14.346559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:54:14.346561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:54:14.346590 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:54:14.348544 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:14.349211 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:14.349215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:14.349217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:14.349312 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:14.349769 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:14.349774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:14.349777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:14.350006 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:54:14.350092 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.350846 (+ 754us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.350987 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 01:54:14.352207 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.336773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:44581 (local address 127.3.220.3:59827)
0809 01:54:14.337370 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:14.337432 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 01:54:14.338573 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:54:14.339189 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:54:14.339198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:54:14.339568 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:54:14.340261 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.340282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.341391 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.341395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.342870 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.342882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.343562 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.343566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.343779 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.344538 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:54:14.344559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:54:14.346362 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:54:14.348701 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:54:14.348708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:54:14.348727 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:54:14.349083 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:54:14.349422 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:54:14.349428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:54:14.349431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:54:14.349639 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:54:14.350162 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:54:14.350169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:54:14.350514 (+ 345us) client_negotiation.cc:769] Sending connection context
0809 01:54:14.350741 (+ 227us) client_negotiation.cc:240] Negotiation successful
0809 01:54:14.351092 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":163,"threads_started":1}
I20260809 01:54:14.353477 20990 heartbeater.cc:347] Connected to a master server at 127.3.220.62:44581
I20260809 01:54:14.353780 20990 heartbeater.cc:464] Registering TS with master...
I20260809 01:54:14.354377 20990 heartbeater.cc:511] Master 127.3.220.62:44581 requested a full tablet report, sending...
I20260809 01:54:14.355558 20238 ts_manager.cc:194] Registered new tserver with Master: d1743ee3932546c6aeb9feb4f87c7893 (127.3.220.3:44653)
I20260809 01:54:14.356199 20238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:59827
I20260809 01:54:14.362058 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:54:14.378813 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.368501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.3.220.62:44581)
0809 01:54:14.368674 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:14.368679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:14.369158 (+ 479us) server_negotiation.cc:407] Connection header received
0809 01:54:14.369302 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:14.369305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:14.369356 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:14.369441 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:54:14.370474 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.371364 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.372687 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.373021 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.373140 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.374979 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:54:14.374983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:54:14.374986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:54:14.375013 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:54:14.376870 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:14.377386 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:14.377390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:14.377392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:14.377484 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:54:14.377844 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:54:14.377848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:54:14.377850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:54:14.378079 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:54:14.378219 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.378504 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.378642 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 01:54:14.382596 20238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:54:14.387259 20238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:54:14.426410 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.417315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43305 (local address 127.0.0.1:42960)
0809 01:54:14.418925 (+ 1610us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:14.418937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:54:14.419023 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:54:14.419824 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:54:14.419830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:54:14.419850 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:54:14.420150 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.420157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.422112 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.422115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.424388 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.424400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.425799 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.425803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.425930 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.425972 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 01:54:14.426092 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 01:54:14.426242 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1550,"mutex_wait_us":25,"thread_start_us":108,"threads_started":1}
I20260809 01:54:14.427047 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.417809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.3.220.1:43305)
0809 01:54:14.418858 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:14.418863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:14.419034 (+ 171us) server_negotiation.cc:407] Connection header received
0809 01:54:14.419122 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:14.419128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:14.419368 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:14.419600 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:54:14.420836 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.421962 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.424549 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.425420 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.425556 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.425726 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.426781 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.426898 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":91,"threads_started":1}
I20260809 01:54:14.430456 20473 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4aa46e7cfa41a085fffc81c8cccd77 (DEFAULT_TABLE table=test-table [id=7263ee5fb6734a6996281f2067198eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:54:14.431566 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.416948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:41775 (local address 127.0.0.1:54538)
0809 01:54:14.418428 (+ 1480us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:14.418483 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 01:54:14.418695 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:54:14.419921 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:54:14.419924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:54:14.419935 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:54:14.420033 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.420039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.428593 (+ 8554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.428596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.429747 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.429760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.431128 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.431133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.431239 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.431264 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 01:54:14.431339 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 01:54:14.431426 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260809 01:54:14.431782 20473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4aa46e7cfa41a085fffc81c8cccd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:54:14.432137 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.417106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.3.220.2:41775)
0809 01:54:14.419422 (+ 2316us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:14.419440 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:14.419466 (+ 26us) server_negotiation.cc:407] Connection header received
0809 01:54:14.419541 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:14.419546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:14.419731 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:14.419888 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:54:14.420437 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.428439 (+ 8002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.429930 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.430802 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.430909 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.431057 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.431896 (+ 839us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.432004 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2216,"thread_start_us":126,"threads_started":1}
I20260809 01:54:14.433744 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.417820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:44653 (local address 127.0.0.1:49836)
0809 01:54:14.419000 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:14.419015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:54:14.419117 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:54:14.419740 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:54:14.419747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:54:14.419771 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:54:14.419964 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.419973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.424978 (+ 5005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.424982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.426272 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.426283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.433304 (+ 7021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.433308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.433423 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.433449 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 01:54:14.433499 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 01:54:14.433585 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":19,"thread_start_us":97,"threads_started":1}
I20260809 01:54:14.435539 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.417990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.3.220.3:44653)
0809 01:54:14.418516 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:14.418536 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:14.419216 (+ 680us) server_negotiation.cc:407] Connection header received
0809 01:54:14.419304 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:14.419309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:14.419505 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:14.420203 (+ 698us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:54:14.420338 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.421608 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.431476 (+ 9868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.432802 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.435045 (+ 2243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.435215 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.435277 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.435385 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":174,"threads_started":1}
I20260809 01:54:14.435312 20684 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4aa46e7cfa41a085fffc81c8cccd77 (DEFAULT_TABLE table=test-table [id=7263ee5fb6734a6996281f2067198eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:54:14.436460 20684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4aa46e7cfa41a085fffc81c8cccd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:54:14.440115 20890 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4aa46e7cfa41a085fffc81c8cccd77 (DEFAULT_TABLE table=test-table [id=7263ee5fb6734a6996281f2067198eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:54:14.441275 20890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4aa46e7cfa41a085fffc81c8cccd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:54:14.443800 21021 tablet_bootstrap.cc:492] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a: Bootstrap starting.
I20260809 01:54:14.446328 21021 tablet_bootstrap.cc:654] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a: Neither blocks nor log segments found. Creating new log.
I20260809 01:54:14.447327 21021 log.cc:824] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a: Log is configured to *not* fsync() on all Append() calls
I20260809 01:54:14.449779 21021 tablet_bootstrap.cc:492] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a: No bootstrap required, opened a new log
I20260809 01:54:14.449988 21021 ts_tablet_manager.cc:1397] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
Aug 09 01:54:14 dist-test-slave-gkw7 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:54:14.454595 21021 raft_consensus.cc:348] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.454881 21021 raft_consensus.cc:374] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:54:14.454931 21021 raft_consensus.cc:729] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d9d1f14e2a4fc0b41e92829043f40a, State: Initialized, Role: FOLLOWER
I20260809 01:54:14.455683 21021 consensus_queue.cc:260] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.456929 21021 ts_tablet_manager.cc:1428] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 01:54:14.457355 20580 heartbeater.cc:503] Master 127.3.220.62:44581 was elected leader, sending a full tablet report...
I20260809 01:54:14.462777 21024 tablet_bootstrap.cc:492] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Bootstrap starting.
I20260809 01:54:14.465912 21024 tablet_bootstrap.cc:654] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Neither blocks nor log segments found. Creating new log.
I20260809 01:54:14.466856 21024 log.cc:824] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Log is configured to *not* fsync() on all Append() calls
I20260809 01:54:14.467422 21025 tablet_bootstrap.cc:492] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: Bootstrap starting.
I20260809 01:54:14.469238 21024 tablet_bootstrap.cc:492] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: No bootstrap required, opened a new log
I20260809 01:54:14.469465 21024 ts_tablet_manager.cc:1397] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 01:54:14.470631 21025 tablet_bootstrap.cc:654] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: Neither blocks nor log segments found. Creating new log.
I20260809 01:54:14.471594 21025 log.cc:824] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: Log is configured to *not* fsync() on all Append() calls
I20260809 01:54:14.473608 21024 raft_consensus.cc:348] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.473915 21024 raft_consensus.cc:374] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:54:14.473989 21024 raft_consensus.cc:729] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1743ee3932546c6aeb9feb4f87c7893, State: Initialized, Role: FOLLOWER
I20260809 01:54:14.474526 21024 consensus_queue.cc:260] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.475687 21024 ts_tablet_manager.cc:1428] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:54:14.476069 20990 heartbeater.cc:503] Master 127.3.220.62:44581 was elected leader, sending a full tablet report...
I20260809 01:54:14.481315 21025 tablet_bootstrap.cc:492] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: No bootstrap required, opened a new log
I20260809 01:54:14.481611 21025 ts_tablet_manager.cc:1397] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260809 01:54:14.486038 21025 raft_consensus.cc:348] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.486352 21025 raft_consensus.cc:374] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:54:14.486411 21025 raft_consensus.cc:729] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca2145750f0402684242c3f09dc19fc, State: Initialized, Role: FOLLOWER
I20260809 01:54:14.486991 21025 consensus_queue.cc:260] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.488287 21025 ts_tablet_manager.cc:1428] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 01:54:14.488716 20792 heartbeater.cc:503] Master 127.3.220.62:44581 was elected leader, sending a full tablet report...
W20260809 01:54:14.521711 20582 tablet.cc:2369] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 01:54:14.589061 20991 tablet.cc:2369] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:54:14.624960 21035 raft_consensus.cc:482] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:54:14.625226 21035 raft_consensus.cc:504] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.626777 21035 leader_election.cc:288] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1743ee3932546c6aeb9feb4f87c7893 (127.3.220.3:44653), 21d9d1f14e2a4fc0b41e92829043f40a (127.3.220.1:43305)
I20260809 01:54:14.645113 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.628450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:56719 (local address 127.3.220.1:43305)
0809 01:54:14.628599 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:14.628604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:14.636432 (+ 7828us) server_negotiation.cc:407] Connection header received
0809 01:54:14.636588 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:14.636592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:14.636651 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:14.636752 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:54:14.638660 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.639416 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.642531 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.643162 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.643248 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.643323 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.644863 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.644968 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 01:54:14.645663 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.628056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43305 (local address 127.3.220.2:56719)
0809 01:54:14.636326 (+ 8270us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:14.636341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:54:14.636482 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:54:14.637330 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:54:14.637333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:54:14.637352 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:54:14.637466 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.637474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.641122 (+ 3648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.641125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.642377 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.642389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.643428 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.643431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.643542 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.643567 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 01:54:14.645379 (+ 1812us) client_negotiation.cc:240] Negotiation successful
0809 01:54:14.645458 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8201,"thread_start_us":101,"threads_started":1}
I20260809 01:54:14.646348 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.627672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:51521 (local address 127.3.220.3:44653)
0809 01:54:14.627815 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:14.627820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:14.636109 (+ 8289us) server_negotiation.cc:407] Connection header received
0809 01:54:14.636749 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:14.636752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:14.636813 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:14.636891 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:54:14.637651 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.638461 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.643713 (+ 5252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.644331 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.644420 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.644485 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.646135 (+ 1650us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.646219 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 01:54:14.646395 20510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4aa46e7cfa41a085fffc81c8cccd77" candidate_uuid: "5ca2145750f0402684242c3f09dc19fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" is_pre_election: true
I20260809 01:54:14.646761 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.627245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:44653 (local address 127.3.220.2:51521)
0809 01:54:14.635972 (+ 8727us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:14.635992 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 01:54:14.636611 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:54:14.637061 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:54:14.637064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:54:14.637083 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:54:14.637207 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.637215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.639579 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.639582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.640935 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:54:14.640947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.644588 (+ 3641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.644591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:54:14.644687 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.644711 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 01:54:14.646547 (+ 1836us) client_negotiation.cc:240] Negotiation successful
0809 01:54:14.646626 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8626,"thread_start_us":92,"threads_started":1}
I20260809 01:54:14.646764 20510 raft_consensus.cc:2454] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ca2145750f0402684242c3f09dc19fc in term 0.
I20260809 01:54:14.647403 20895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4aa46e7cfa41a085fffc81c8cccd77" candidate_uuid: "5ca2145750f0402684242c3f09dc19fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1743ee3932546c6aeb9feb4f87c7893" is_pre_election: true
I20260809 01:54:14.647472 20617 leader_election.cc:302] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d9d1f14e2a4fc0b41e92829043f40a, 5ca2145750f0402684242c3f09dc19fc; no voters:
I20260809 01:54:14.647704 20895 raft_consensus.cc:2454] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ca2145750f0402684242c3f09dc19fc in term 0.
I20260809 01:54:14.647837 21035 raft_consensus.cc:2781] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:54:14.647943 21035 raft_consensus.cc:482] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:54:14.648003 21035 raft_consensus.cc:3037] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:54:14.649433 21035 raft_consensus.cc:504] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.649981 21035 leader_election.cc:288] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [CANDIDATE]: Term 1 election: Requested vote from peers d1743ee3932546c6aeb9feb4f87c7893 (127.3.220.3:44653), 21d9d1f14e2a4fc0b41e92829043f40a (127.3.220.1:43305)
I20260809 01:54:14.650532 20895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4aa46e7cfa41a085fffc81c8cccd77" candidate_uuid: "5ca2145750f0402684242c3f09dc19fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1743ee3932546c6aeb9feb4f87c7893"
I20260809 01:54:14.650655 20895 raft_consensus.cc:3037] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:54:14.650770 20510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4aa46e7cfa41a085fffc81c8cccd77" candidate_uuid: "5ca2145750f0402684242c3f09dc19fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d9d1f14e2a4fc0b41e92829043f40a"
I20260809 01:54:14.650898 20510 raft_consensus.cc:3037] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:54:14.651984 20895 raft_consensus.cc:2454] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ca2145750f0402684242c3f09dc19fc in term 1.
I20260809 01:54:14.652148 20510 raft_consensus.cc:2454] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ca2145750f0402684242c3f09dc19fc in term 1.
I20260809 01:54:14.652457 20613 leader_election.cc:302] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ca2145750f0402684242c3f09dc19fc, d1743ee3932546c6aeb9feb4f87c7893; no voters:
I20260809 01:54:14.652726 21035 raft_consensus.cc:2781] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:54:14.653014 21035 raft_consensus.cc:686] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 1 LEADER]: Becoming Leader. State: Replica: 5ca2145750f0402684242c3f09dc19fc, State: Running, Role: LEADER
I20260809 01:54:14.653420 21035 consensus_queue.cc:237] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } }
I20260809 01:54:14.657114 20237 catalog_manager.cc:5495] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ca2145750f0402684242c3f09dc19fc (127.3.220.2). New cstate: current_term: 1 leader_uuid: "5ca2145750f0402684242c3f09dc19fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ca2145750f0402684242c3f09dc19fc" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 } health_report { overall_health: UNKNOWN } } }
W20260809 01:54:14.715247 20793 tablet.cc:2369] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:54:14.715880 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.710346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.3.220.2:41775)
0809 01:54:14.710714 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0809 01:54:14.710719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:54:14.710734 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:54:14.710840 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:54:14.710843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:54:14.710926 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:54:14.711041 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:54:14.711401 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.712209 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.713505 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:54:14.713903 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:54:14.714013 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:54:14.714458 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:54:14.715082 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:54:14.715209 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 01:54:14.715676 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 01:54:14.715742 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260809 01:54:14.730525 20510 raft_consensus.cc:1261] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 21d9d1f14e2a4fc0b41e92829043f40a [term 1 FOLLOWER]: Refusing update from remote peer 5ca2145750f0402684242c3f09dc19fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:54:14.731436 21035 consensus_queue.cc:1035] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d9d1f14e2a4fc0b41e92829043f40a" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:54:14.732270 20895 raft_consensus.cc:1261] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 1 FOLLOWER]: Refusing update from remote peer 5ca2145750f0402684242c3f09dc19fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:54:14.732957 21035 consensus_queue.cc:1035] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1743ee3932546c6aeb9feb4f87c7893" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:54:14.740971 21045 mvcc.cc:205] Tried to move back new op lower bound from 7316440902565396480 to 7316440902264262656. Current Snapshot: MvccSnapshot[applied={T|T < 7316440902565396480}]
I20260809 01:54:14.746932 21057 mvcc.cc:205] Tried to move back new op lower bound from 7316440902565396480 to 7316440902264262656. Current Snapshot: MvccSnapshot[applied={T|T < 7316440902565396480}]
I20260809 01:54:14.747211 21058 mvcc.cc:205] Tried to move back new op lower bound from 7316440902565396480 to 7316440902264262656. Current Snapshot: MvccSnapshot[applied={T|T < 7316440902565396480}]
I20260809 01:54:14.752996 20237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:54:14.753197 20237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:54:14.756502 20237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50c0e627a91f45b3bdf31d85b6888408: Sending DeleteTablet for 3 replicas of tablet 8d4aa46e7cfa41a085fffc81c8cccd77
I20260809 01:54:14.757654 20473 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4aa46e7cfa41a085fffc81c8cccd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:54:14 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:42960
I20260809 01:54:14.758741 20684 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4aa46e7cfa41a085fffc81c8cccd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:54:14 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:54538
I20260809 01:54:14.759227 20890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4aa46e7cfa41a085fffc81c8cccd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:54:14 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:49836
I20260809 01:54:14.760380 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 20368
I20260809 01:54:14.769765 21074 tablet_replica.cc:330] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: stopping tablet replica
I20260809 01:54:14.770184 21074 raft_consensus.cc:2229] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:54:14.770660 21074 raft_consensus.cc:2258] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:54:14.771092 21075 tablet_replica.cc:330] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: stopping tablet replica
I20260809 01:54:14.771466 21075 raft_consensus.cc:2229] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:54:14.771778 21075 raft_consensus.cc:2258] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:54:14.773674 21074 ts_tablet_manager.cc:1905] T 8d4aa46e7cfa41a085fffc81c8cccd77 P 5ca2145750f0402684242c3f09dc19fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:54:14.776207 21075 ts_tablet_manager.cc:1905] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:54:14.779217 21075 ts_tablet_manager.cc:1918] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:54:14.779335 21075 log.cc:1196] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/8d4aa46e7cfa41a085fffc81c8cccd77
I20260809 01:54:14.779651 21075 ts_tablet_manager.cc:1939] T 8d4aa46e7cfa41a085fffc81c8cccd77 P d1743ee3932546c6aeb9feb4f87c7893: Deleting consensus metadata
I20260809 01:54:14.781033 20219 catalog_manager.cc:4841] TS d1743ee3932546c6aeb9feb4f87c7893 (127.3.220.3:44653): tablet 8d4aa46e7cfa41a085fffc81c8cccd77 (table test-table [id=7263ee5fb6734a6996281f2067198eb9]) successfully deleted
W20260809 01:54:14.781608 20221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:43305 (error 108)
I20260809 01:54:14.782573 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 20590
I20260809 01:54:14.790501 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.790201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43305 (local address 127.0.0.1:42964)
0809 01:54:14.790343 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:14.790422 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 01:54:14.790728 20221 catalog_manager.cc:4584] TS 21d9d1f14e2a4fc0b41e92829043f40a (127.3.220.1:43305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d4aa46e7cfa41a085fffc81c8cccd77: Network error: Client connection negotiation failed: client connection to 127.3.220.1:43305: connect: Connection refused (error 111)
I20260809 01:54:14.797084 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 20797
I20260809 01:54:14.797253 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:54:14.796955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:41775 (local address 127.0.0.1:54560)
0809 01:54:14.797081 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 01:54:14.797138 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.2:41775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 01:54:14.806234 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 20197
I20260809 01:54:14.822338 3952 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 01:54:15.167124 3952 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 01:54:15.167310 3952 mini_ranger.cc:66] Stopping Ranger...
I20260809 01:54:15.522336 3952 mini_ranger.cc:68] Stopped Ranger
2026-08-09 01:54:15.522 UTC [19578] LOG: received smart shutdown request
2026-08-09 01:54:15.526 UTC [19578] LOG: background worker "logical replication launcher" (PID 19590) exited with exit code 1
2026-08-09 01:54:15.526 UTC [19582] LOG: shutting down
2026-08-09 01:54:15.692 UTC [19578] LOG: database system is shut down
2026-08-09T01:54:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48337 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:54:15 dist-test-slave-gkw7 krb5kdc[21090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:54:15 dist-test-slave-gkw7 krb5kdc[21090](info): set up 2 sockets
Aug 09 01:54:15 dist-test-slave-gkw7 krb5kdc[21090](info): commencing operation
krb5kdc: starting...
W20260809 01:54:17.064533 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:54:17 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:54:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.220.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.220.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/rangeradmin_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/rangeradmin_127.3.220.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.220.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.220.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/rangerlookup_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/rangerlookup_127.3.220.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.220.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.220.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/HTTP_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/HTTP_127.3.220.20@KRBTEST.COM.keytab.
I20260809 01:54:17.244596 3952 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIuGTYa/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/postgres -l logfile start
2026-08-09 01:54:19.822 UTC [21132] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 01:54:19.822 UTC [21132] LOG: listening on IPv4 address "127.3.220.20", port 48101
2026-08-09 01:54:19.826 UTC [21132] LOG: listening on Unix socket "/tmp/.s.PGSQL.48101"
2026-08-09 01:54:19.848 UTC [21137] LOG: database system was shut down at 2026-08-09 01:54:18 UTC
2026-08-09 01:54:19.852 UTC [21132] LOG: database system is ready to accept connections
I20260809 01:54:21.002107 3952 mini_postgres.cc:96] Postgres bound to 48101
2026-08-09 01:54:21.009 UTC [21148] FATAL: database "slave" does not exist
127.3.220.20:48101 - accepting connections
I20260809 01:54:21.010213 3952 mini_ranger.cc:161] Starting Ranger...
I20260809 01:54:21.026346 3952 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 01:54:21.518591 3952 mini_ranger.cc:87] Created ranger Postgres database
I20260809 01:54:21.518687 3952 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-admin
2026-08-09 01:54:21,795 [I] DB FLAVOR :POSTGRES
2026-08-09 01:54:21,795 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 01:54:21,795 [I] Checking connection..
2026-08-09 01:54:21,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 01:54:22,074 [I] Checking connection passed.
2026-08-09 01:54:22,075 [I] --------- Verifying version history table ---------
2026-08-09 01:54:22,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 01:54:22,335 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 01:54:22,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 01:54:22,593 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 01:54:22,593 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 01:54:22,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 01:54:22.857 UTC [21239] WARNING: there is no transaction in progress
2026-08-09 01:54:22,865 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 01:54:22,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 01:54:23,140 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 01:54:23,141 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 01:54:23,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 01:54:23,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 01:54:23,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gkw7.c.gcp-upstream.internal','N') ;"
2026-08-09 01:54:23,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 01:54:24,197 [I] Table x_portal_user does not exist in database ranger
2026-08-09 01:54:24,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 01:54:24,468 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 01:54:24,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 01:54:24,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 01:54:24,989 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 01:54:24,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 01:54:25.964 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:25.982 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.000 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.016 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.033 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.053 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.118 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.125 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.139 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.149 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.160 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.175 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.185 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.192 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.199 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26.207 UTC [21397] WARNING: there is no transaction in progress
2026-08-09 01:54:26,850 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 01:54:26,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 01:54:27,113 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 01:54:27,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 01:54:27,360 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 01:54:27,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 01:54:27,641 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 01:54:27,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 01:54:27,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 01:54:28,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gkw7.c.gcp-upstream.internal';"
2026-08-09 01:54:28,571 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 01:54:28,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 01:54:28,825 [I] DB_PATCHES have already been applied
I20260809 01:54:28.831629 3952 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIuGTYa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 01:54:28.831712 3952 mini_ranger.cc:193] Using host: 127.3.220.20
I20260809 01:54:28.836464 3952 mini_ranger.cc:227] Ranger admin URL: http://127.3.220.20:34771
Aug 09, 2026 1:54:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 1:54:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 1:54:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 1:54:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 1:54:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 1:54:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 1:54:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 01:54:30.264973 3952 mini_ranger.cc:160] Time spent starting Ranger: real 9.255s user 0.000s sys 0.012s
Aug 09, 2026 1:54:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 1:54:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 09, 2026 1:54:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 1:54:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 1:54:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 01:54:53.191956 3952 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.220.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.220.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/rangerkms_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.220.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/rangerkms_127.3.220.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.220.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.220.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.220.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/HTTP_127.3.220.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.220.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/HTTP_127.3.220.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 01:54:53 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240493, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 01:54:53.302161 3952 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 01:54:53.313755 3952 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 01:54:54.237141 3952 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 01:54:54.237259 3952 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms
I20260809 01:54:54.237291 3952 mini_ranger_kms.cc:227] Using postgres at 127.3.220.20:48101
2026-08-09 01:54:54,428 [I] DB FLAVOR :POSTGRES
2026-08-09 01:54:54,428 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 01:54:54,428 [I] Checking connection
2026-08-09 01:54:54,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 01:54:54,726 [I] connection success
2026-08-09 01:54:54,726 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 01:54:54,726 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 01:54:54,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 01:54:54,994 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 01:54:54,995 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 01:54:54,995 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 01:54:54,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.220.20:48101/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 01:54:55,299 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 01:54:55.780835 3952 mini_ranger_kms.cc:310] Created kms service
I20260809 01:54:56.022640 3952 mini_ranger_kms.cc:326] Created kudu user
I20260809 01:54:56.072126 3952 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 01:54:56 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240493, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:54:56.492973 3952 mini_ranger_kms.cc:384] Added ranger policy
I20260809 01:54:56.493224 3952 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIuGTYa/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIuGTYa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 01:54:56.493249 3952 mini_ranger_kms.cc:240] Using host: 127.3.220.20
I20260809 01:54:56.498204 3952 mini_ranger_kms.cc:276] Ranger KMS PID: 21669
I20260809 01:54:56.498309 3952 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.220.20:53681
Aug 09, 2026 1:54:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 1:54:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 1:54:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 1:54:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 1:54:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 1:54:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 01:54:59 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786240499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.220.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:54:59.322984 3952 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.021s user 0.013s sys 0.162s
I20260809 01:54:59.323135 3952 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 01:54:59.323215 3952 mini_ranger_kms.cc:401] 127.3.220.20:53681/kms/v1/keys
Aug 09 01:54:59 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.220.20@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
Aug 09 01:55:00 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.220.20@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
Aug 09 01:55:02 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:55:02.323899 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41249
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:41249
--ranger_config_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:53681/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:02.453461 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:02.453791 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:02.453832 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:02.457953 21744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:02.458012 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:02.458032 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:02.458050 21744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:02.458065 21744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:02.463742 21744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.220.20:53681/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:41249
--ranger_config_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:02.464838 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:02.465948 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:02.475330 21754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:02.475448 21750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:02.475395 21744 server_base.cc:1034] running on GCE node
W20260809 01:55:02.475344 21756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:02.476264 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:02.477779 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:02.478992 21744 hybrid_clock.cc:648] HybridClock initialized: now 1786240502478957 us; error 45 us; skew 500 ppm
Aug 09 01:55:02 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240502, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:02.482549 21744 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:02.483518 21744 webserver.cc:460] Webserver started at http://127.3.220.62:35207/ using document root <none> and password file <none>
I20260809 01:55:02.484088 21744 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:02.484148 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:02.484372 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 01:55:02 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240502, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:02.539681 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "e501253e2dfc4235bff3972fccaa8c0d"
format_stamp: "Formatted at 2026-08-09 01:55:02 on dist-test-slave-gkw7"
server_key: "40cdd7a47b480f188d166b5ad18121f7"
server_key_iv: "b8c6c509fe832345e619c0b97d6f9201"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:02.540402 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "e501253e2dfc4235bff3972fccaa8c0d"
format_stamp: "Formatted at 2026-08-09 01:55:02 on dist-test-slave-gkw7"
server_key: "40cdd7a47b480f188d166b5ad18121f7"
server_key_iv: "b8c6c509fe832345e619c0b97d6f9201"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:02.545640 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260809 01:55:02.561818 21764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:02.563476 21744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:55:02.563653 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "e501253e2dfc4235bff3972fccaa8c0d"
format_stamp: "Formatted at 2026-08-09 01:55:02 on dist-test-slave-gkw7"
server_key: "40cdd7a47b480f188d166b5ad18121f7"
server_key_iv: "b8c6c509fe832345e619c0b97d6f9201"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:02.563802 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:02.601805 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:02.605456 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:02.605742 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:02.615283 21744 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:41249
I20260809 01:55:02.615304 21830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:41249 every 8 connection(s)
I20260809 01:55:02.616463 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:02.619655 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:02.624042 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 21744
I20260809 01:55:02.624185 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:55:02.624508 3952 external_mini_cluster.cc:1428] Setting key 6ae7fd8e51622532a73c4170fbab0bdd
I20260809 01:55:02.626395 21832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Bootstrap starting.
I20260809 01:55:02.629436 21832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:02.630528 21832 log.cc:824] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:02.632915 21832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: No bootstrap required, opened a new log
Aug 09 01:55:02 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:02.637794 21832 raft_consensus.cc:348] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:55:02.638095 21832 raft_consensus.cc:374] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:02.638198 21832 raft_consensus.cc:729] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e501253e2dfc4235bff3972fccaa8c0d, State: Initialized, Role: FOLLOWER
I20260809 01:55:02.638736 21832 consensus_queue.cc:260] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:55:02.638939 21832 raft_consensus.cc:388] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:55:02.639017 21832 raft_consensus.cc:482] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:55:02.639150 21832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:02.640337 21832 raft_consensus.cc:504] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:55:02.640769 21832 leader_election.cc:302] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e501253e2dfc4235bff3972fccaa8c0d; no voters:
I20260809 01:55:02.641165 21832 leader_election.cc:288] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:55:02.641323 21839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:02.641705 21839 raft_consensus.cc:686] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 1 LEADER]: Becoming Leader. State: Replica: e501253e2dfc4235bff3972fccaa8c0d, State: Running, Role: LEADER
I20260809 01:55:02.642118 21839 consensus_queue.cc:237] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:55:02.642197 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:02.626192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.3.220.62:41249)
0809 01:55:02.626757 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:02.626779 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:02.626827 (+ 48us) server_negotiation.cc:407] Connection header received
0809 01:55:02.627805 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:02.627831 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:02.628310 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:02.628743 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:02.629705 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:02.631301 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:02.632313 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:02.632845 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:02.633035 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:02.635981 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:02.635987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:02.636003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:02.636034 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:02.638055 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:02.638677 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:02.638682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:02.638687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:02.638799 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:02.639360 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:02.639364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:02.639365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:02.639808 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:02.640054 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:02.640378 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 01:55:02.640904 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":155,"threads_started":1}
I20260809 01:55:02.642690 21832 sys_catalog.cc:564] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:02.643572 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e501253e2dfc4235bff3972fccaa8c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } } }
I20260809 01:55:02.643703 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:02.643908 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e501253e2dfc4235bff3972fccaa8c0d. Latest consensus state: current_term: 1 leader_uuid: "e501253e2dfc4235bff3972fccaa8c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } } }
I20260809 01:55:02.643983 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:02.645090 21832 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 01:55:02.645736 21853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:02.653965 21853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:02.664980 21853 catalog_manager.cc:1347] Generated new cluster ID: 745ef6a6f9684f938d8f70b05dfcae1d
I20260809 01:55:02.665064 21853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:55:02.718590 21853 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:55:02.720263 21853 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:55:02.735018 21853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Generated new TSK 0
I20260809 01:55:02.736200 21853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:55:03.526566 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:53681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:03.672757 21896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:03.673105 21896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:03.673177 21896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:03.679208 21896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:03.679283 21896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:03.679379 21896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:55:03.685943 21896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.220.20:53681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:03.687350 21896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:03.688709 21896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:03.701131 21907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:03.701468 21896 server_base.cc:1034] running on GCE node
W20260809 01:55:03.701480 21909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:03.701828 21906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:03.702376 21896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:03.703176 21896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:03.704373 21896 hybrid_clock.cc:648] HybridClock initialized: now 1786240503704352 us; error 37 us; skew 500 ppm
Aug 09 01:55:03 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:03.707432 21896 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:55:03.708490 21896 webserver.cc:460] Webserver started at http://127.3.220.1:43823/ using document root <none> and password file <none>
I20260809 01:55:03.709097 21896 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:03.709203 21896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:03.709522 21896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 01:55:03 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:03.745347 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "88dac609f00840f69c390a234fbbd889"
format_stamp: "Formatted at 2026-08-09 01:55:03 on dist-test-slave-gkw7"
server_key: "9ab3dec21489e336e400205834cf02f2"
server_key_iv: "b6fafa0201d52d695f9b1024b01b5090"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:03.746250 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "88dac609f00840f69c390a234fbbd889"
format_stamp: "Formatted at 2026-08-09 01:55:03 on dist-test-slave-gkw7"
server_key: "9ab3dec21489e336e400205834cf02f2"
server_key_iv: "b6fafa0201d52d695f9b1024b01b5090"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:03.751704 21896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 01:55:03.773967 21923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:03.775803 21896 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260809 01:55:03.776026 21896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "88dac609f00840f69c390a234fbbd889"
format_stamp: "Formatted at 2026-08-09 01:55:03 on dist-test-slave-gkw7"
server_key: "9ab3dec21489e336e400205834cf02f2"
server_key_iv: "b6fafa0201d52d695f9b1024b01b5090"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:03.776278 21896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:03.798709 21896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:03.802578 21896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:03.802841 21896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:03.803618 21896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:03.804719 21896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:03.804840 21896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:03.804965 21896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:03.805054 21896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:03.818658 21896 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:32821
I20260809 01:55:03.819916 21896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:55:03.821273 22082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:32821 every 8 connection(s)
I20260809 01:55:03.826308 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 21896
I20260809 01:55:03.826444 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:55:03.826797 3952 external_mini_cluster.cc:1428] Setting key b099f4e83ea3c91cce2a0a721ee528d8
Aug 09 01:55:03 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:55:03.841970 22086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:03.823861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:36041 (local address 127.3.220.62:41249)
0809 01:55:03.824142 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:03.824147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:03.825334 (+ 1187us) server_negotiation.cc:407] Connection header received
0809 01:55:03.826699 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:03.826702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:03.826766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:03.826866 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:03.828388 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:03.829324 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:03.831101 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:03.831436 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:03.831561 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:03.837067 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:03.837071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:03.837074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:03.837103 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:03.839151 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:03.839845 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:03.839848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:03.839850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:03.839947 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:03.840393 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:03.840396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:03.840397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:03.840713 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:03.840796 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:03.841475 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 01:55:03.841731 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260809 01:55:03.842881 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:03.823928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.1:36041)
0809 01:55:03.825112 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:03.825185 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:03.826336 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:03.827145 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:03.827155 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:03.827526 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:03.828193 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:03.828213 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:03.829490 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:03.829496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:03.830960 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:03.830972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:03.831689 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:03.831697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:03.831931 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:03.834748 (+ 2817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:03.834769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:03.836828 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:03.839296 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:03.839304 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:03.839322 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:03.839718 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:03.840074 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:03.840077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:03.840079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:03.840267 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:03.840833 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:03.840840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:03.841176 (+ 336us) client_negotiation.cc:769] Sending connection context
0809 01:55:03.841400 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 01:55:03.841800 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":162,"threads_started":1}
Aug 09 01:55:03 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786240503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:55:03.844538 22083 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:55:03.844877 22083 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:03.845469 22083 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:55:03.849110 21786 ts_manager.cc:194] Registered new tserver with Master: 88dac609f00840f69c390a234fbbd889 (127.3.220.1:32821)
I20260809 01:55:03.850762 21786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:36041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:55:03.865051 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:53681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:04.022809 22090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:04.023239 22090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:04.023367 22090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:04.029515 22090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:04.029709 22090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:04.029844 22090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:55:04.037742 22090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.220.20:53681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:04.039320 22090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:04.040691 22090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:04.055286 22100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:04.056217 22098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:04.056375 22097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:04.056921 22090 server_base.cc:1034] running on GCE node
I20260809 01:55:04.057406 22090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:04.058226 22090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:04.059435 22090 hybrid_clock.cc:648] HybridClock initialized: now 1786240504059408 us; error 41 us; skew 500 ppm
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:04.063012 22090 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:55:04.063958 22090 webserver.cc:460] Webserver started at http://127.3.220.2:34175/ using document root <none> and password file <none>
I20260809 01:55:04.064595 22090 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:04.064688 22090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:04.064924 22090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:04.080264 22090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "39a35a78693c44cc83cc67b496d4d2ad"
format_stamp: "Formatted at 2026-08-09 01:55:04 on dist-test-slave-gkw7"
server_key: "7d06c063bca541e8eb4b876f903eb970"
server_key_iv: "c3ae444f5dcbe0d71fb97a4e42790ab6"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:04.080999 22090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "39a35a78693c44cc83cc67b496d4d2ad"
format_stamp: "Formatted at 2026-08-09 01:55:04 on dist-test-slave-gkw7"
server_key: "7d06c063bca541e8eb4b876f903eb970"
server_key_iv: "c3ae444f5dcbe0d71fb97a4e42790ab6"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:04.086711 22090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 01:55:04.105562 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:04.107409 22090 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260809 01:55:04.107555 22090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "39a35a78693c44cc83cc67b496d4d2ad"
format_stamp: "Formatted at 2026-08-09 01:55:04 on dist-test-slave-gkw7"
server_key: "7d06c063bca541e8eb4b876f903eb970"
server_key_iv: "c3ae444f5dcbe0d71fb97a4e42790ab6"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:04.107705 22090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:04.128558 22090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:04.132401 22090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:04.132695 22090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:04.133430 22090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:04.134598 22090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:04.134727 22090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:04.134836 22090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:04.134903 22090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:04.146095 22090 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:38279
I20260809 01:55:04.147322 22090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:55:04.153187 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:38279 every 8 connection(s)
I20260809 01:55:04.156047 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 22090
I20260809 01:55:04.156195 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:55:04.156553 3952 external_mini_cluster.cc:1428] Setting key 572cea49968f6bc2c161ad45ba14935a
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:55:04.177043 22086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:04.157252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:38411 (local address 127.3.220.62:41249)
0809 01:55:04.157385 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:04.157390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:04.157680 (+ 290us) server_negotiation.cc:407] Connection header received
0809 01:55:04.161565 (+ 3885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:04.161569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:04.161634 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:04.161692 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:04.163444 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:04.164307 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:04.167199 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:04.167550 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:04.167650 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:04.171657 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:04.171661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:04.171664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:04.171693 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:04.173563 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:04.174267 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:04.174271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:04.174272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:04.174368 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:04.174834 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:04.174838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:04.174840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:04.175264 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:04.175365 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:04.176692 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0809 01:55:04.176839 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 01:55:04.178840 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:04.156091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.2:38411)
0809 01:55:04.157463 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:04.157522 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:04.161254 (+ 3732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:04.162156 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:04.162172 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:04.162546 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:04.163259 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:04.163281 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:04.165517 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:04.165521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:04.167042 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:04.167055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:04.167871 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:04.167875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:04.168075 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:04.169690 (+ 1615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:04.169712 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:04.171454 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:04.173676 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:04.173685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:04.173703 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:04.174126 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:04.174502 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:04.174506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:04.174508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:04.174706 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:04.175443 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:04.175450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:04.176372 (+ 922us) client_negotiation.cc:769] Sending connection context
0809 01:55:04.177306 (+ 934us) client_negotiation.cc:240] Negotiation successful
0809 01:55:04.177703 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"thread_start_us":163,"threads_started":1}
I20260809 01:55:04.180649 22269 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:55:04.180928 22269 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:04.181600 22269 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:55:04.182940 21786 ts_manager.cc:194] Registered new tserver with Master: 39a35a78693c44cc83cc67b496d4d2ad (127.3.220.2:38279)
I20260809 01:55:04.183732 21786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:38411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:55:04.197206 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:53681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
W20260809 01:55:04.366701 22274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:04.367031 22274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:04.367166 22274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:04.373119 22274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:04.373246 22274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:04.373358 22274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:55:04.380744 22274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.220.20:53681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:04.382381 22274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:04.383745 22274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:04.399695 22279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:04.399695 22283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:04.400674 22274 server_base.cc:1034] running on GCE node
W20260809 01:55:04.401717 22280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:04.402292 22274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:04.403163 22274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:04.404420 22274 hybrid_clock.cc:648] HybridClock initialized: now 1786240504404382 us; error 52 us; skew 500 ppm
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:04.407914 22274 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:55:04.409078 22274 webserver.cc:460] Webserver started at http://127.3.220.3:46809/ using document root <none> and password file <none>
I20260809 01:55:04.409818 22274 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:04.409929 22274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:04.410195 22274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:04.433219 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "ad186e9309774e999b580bcd0466afb5"
format_stamp: "Formatted at 2026-08-09 01:55:04 on dist-test-slave-gkw7"
server_key: "6658df923551c32c19f7d662fcc31eb4"
server_key_iv: "34f03a70a8a55adb1f4483a8988145a5"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:04.434187 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "ad186e9309774e999b580bcd0466afb5"
format_stamp: "Formatted at 2026-08-09 01:55:04 on dist-test-slave-gkw7"
server_key: "6658df923551c32c19f7d662fcc31eb4"
server_key_iv: "34f03a70a8a55adb1f4483a8988145a5"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:04.443169 22274 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260809 01:55:04.463181 22290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:04.465157 22274 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260809 01:55:04.465394 22274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "ad186e9309774e999b580bcd0466afb5"
format_stamp: "Formatted at 2026-08-09 01:55:04 on dist-test-slave-gkw7"
server_key: "6658df923551c32c19f7d662fcc31eb4"
server_key_iv: "34f03a70a8a55adb1f4483a8988145a5"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:04.465680 22274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:04.496892 22274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:04.501603 22274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:04.501897 22274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:04.502770 22274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:04.503996 22274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:04.504117 22274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:04.504247 22274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:04.504329 22274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:04.517378 22274 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:46691
I20260809 01:55:04.518878 22274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:55:04.522037 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 22274
I20260809 01:55:04.522189 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:55:04.522553 3952 external_mini_cluster.cc:1428] Setting key 4c72f5b81f7be90633ddfc48d6e9349e
I20260809 01:55:04.527771 22440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:46691 every 8 connection(s)
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:04.553673 22086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:04.533267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:38905 (local address 127.3.220.62:41249)
0809 01:55:04.533402 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:04.533407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:04.534261 (+ 854us) server_negotiation.cc:407] Connection header received
0809 01:55:04.537404 (+ 3143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:04.537408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:04.537468 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:04.537523 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:04.539407 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:04.540359 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:04.542352 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:04.542749 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:04.542844 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:04.546582 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:04.546587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:04.546590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:04.546620 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:04.550617 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:04.551427 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:04.551431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:04.551433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:04.551540 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:04.552092 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:04.552096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:04.552099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:04.552375 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:04.552474 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:04.553310 (+ 836us) server_negotiation.cc:299] Negotiation successful
0809 01:55:04.553457 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:55:04.555883 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:04.531986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.3:38905)
0809 01:55:04.534078 (+ 2092us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:04.534155 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:04.537137 (+ 2982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:04.538023 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:04.538034 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:04.538453 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:04.539210 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:04.539233 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:04.540537 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:04.540540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:04.542161 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:04.542173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:04.543150 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:04.543155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:04.543431 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:04.544230 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:04.544252 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:04.546353 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:04.550823 (+ 4470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:04.550830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:04.550850 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:04.551289 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:04.551680 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:04.551684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:04.551686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:04.551962 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:04.552552 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:04.552559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:04.552951 (+ 392us) client_negotiation.cc:769] Sending connection context
0809 01:55:04.554215 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0809 01:55:04.554641 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1853,"thread_start_us":174,"threads_started":1}
I20260809 01:55:04.557636 22441 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:55:04.557926 22441 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:04.558573 22441 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:55:04.559926 21786 ts_manager.cc:194] Registered new tserver with Master: ad186e9309774e999b580bcd0466afb5 (127.3.220.3:46691)
I20260809 01:55:04.560770 21786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:38905
I20260809 01:55:04.565237 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:55:04.566298 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 21896
I20260809 01:55:04.609498 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 22090
I20260809 01:55:04.641098 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 22274
I20260809 01:55:04.666661 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 21744
I20260809 01:55:04.679912 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41249
--webserver_interface=127.3.220.62
--webserver_port=35207
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:41249
--ranger_config_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:53681/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.220.20:53681/kms with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:04.794376 22447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:04.794626 22447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:04.794664 22447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:04.798722 22447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:04.798782 22447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:04.798801 22447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:04.798817 22447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:04.798831 22447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:04.804265 22447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.220.20:53681/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:41249
--ranger_config_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=35207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:04.805298 22447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:04.806392 22447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:04.815016 22453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:04.815161 22447 server_base.cc:1034] running on GCE node
W20260809 01:55:04.815009 22454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:04.815009 22456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:04.815999 22447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:04.817088 22447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:04.818315 22447 hybrid_clock.cc:648] HybridClock initialized: now 1786240504818285 us; error 52 us; skew 500 ppm
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:04.821058 22447 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:04.822001 22447 webserver.cc:460] Webserver started at http://127.3.220.62:35207/ using document root <none> and password file <none>
I20260809 01:55:04.822542 22447 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:04.822644 22447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 01:55:04 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:04.870741 22447 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 01:55:04.873863 22470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:04.875314 22447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:55:04.875463 22447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "e501253e2dfc4235bff3972fccaa8c0d"
format_stamp: "Formatted at 2026-08-09 01:55:02 on dist-test-slave-gkw7"
server_key: "40cdd7a47b480f188d166b5ad18121f7"
server_key_iv: "b8c6c509fe832345e619c0b97d6f9201"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:04.876003 22447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:04.916720 22447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:04.919641 22447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:04.920205 22447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:04.927403 22447 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:41249
I20260809 01:55:04.927421 22537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:41249 every 8 connection(s)
I20260809 01:55:04.928864 22447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:04.933203 22539 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:55:04.934867 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Bootstrap starting.
I20260809 01:55:04.938179 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 22447
I20260809 01:55:04.938722 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:53681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:32821
--local_ip_for_outbound_sockets=127.3.220.1
--tserver_master_addrs=127.3.220.62:41249
--webserver_port=43823
--webserver_interface=127.3.220.1
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.220.20:53681/kms with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:04.952315 22539 log.cc:824] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:04.959267 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:55:04.959686 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Bootstrap complete.
I20260809 01:55:04.964052 22539 raft_consensus.cc:348] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:55:04.964486 22539 raft_consensus.cc:729] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e501253e2dfc4235bff3972fccaa8c0d, State: Initialized, Role: FOLLOWER
I20260809 01:55:04.964972 22539 consensus_queue.cc:260] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:55:04.965116 22539 raft_consensus.cc:388] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:55:04.965204 22539 raft_consensus.cc:482] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:55:04.965327 22539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:55:04.967242 22539 raft_consensus.cc:504] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:55:04.967573 22539 leader_election.cc:302] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e501253e2dfc4235bff3972fccaa8c0d; no voters:
I20260809 01:55:04.967916 22539 leader_election.cc:288] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 01:55:04.968003 22543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 2 FOLLOWER]: Leader election won for term 2
I20260809 01:55:04.968250 22543 raft_consensus.cc:686] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [term 2 LEADER]: Becoming Leader. State: Replica: e501253e2dfc4235bff3972fccaa8c0d, State: Running, Role: LEADER
I20260809 01:55:04.968592 22543 consensus_queue.cc:237] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } }
I20260809 01:55:04.969177 22539 sys_catalog.cc:564] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:04.970182 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e501253e2dfc4235bff3972fccaa8c0d. Latest consensus state: current_term: 2 leader_uuid: "e501253e2dfc4235bff3972fccaa8c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } } }
I20260809 01:55:04.970317 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:04.970696 22559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:04.970690 22544 sys_catalog.cc:455] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e501253e2dfc4235bff3972fccaa8c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501253e2dfc4235bff3972fccaa8c0d" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41249 } } }
I20260809 01:55:04.970863 22544 sys_catalog.cc:458] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:04.974027 22559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:04.974576 22539 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 01:55:04.974797 22559 catalog_manager.cc:1259] Loaded cluster ID: 745ef6a6f9684f938d8f70b05dfcae1d
I20260809 01:55:04.974838 22559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:55:04.976608 22559 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:55:04.977186 22559 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Loaded TSK: 0
I20260809 01:55:04.980618 22559 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 01:55:05.101647 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:05.102027 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:05.102098 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:05.108762 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:05.108848 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:05.108943 22541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:55:05.117740 22541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.220.20:53681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:32821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=43823
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:05.119349 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:05.120818 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:05.134006 22605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:05.134403 22601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:05.135263 22541 server_base.cc:1034] running on GCE node
W20260809 01:55:05.135568 22600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:05.136165 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:05.137017 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:05.141606 22541 hybrid_clock.cc:648] HybridClock initialized: now 1786240505140827 us; error 786 us; skew 500 ppm
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:05.146885 22541 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:55:05.148139 22541 webserver.cc:460] Webserver started at http://127.3.220.1:43823/ using document root <none> and password file <none>
I20260809 01:55:05.148947 22541 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:05.149118 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:05.191213 22541 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260809 01:55:05.195024 22614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.196974 22541 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 01:55:05.197113 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "88dac609f00840f69c390a234fbbd889"
format_stamp: "Formatted at 2026-08-09 01:55:03 on dist-test-slave-gkw7"
server_key: "9ab3dec21489e336e400205834cf02f2"
server_key_iv: "b6fafa0201d52d695f9b1024b01b5090"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:05.198057 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:05.227872 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:05.231328 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:05.231561 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:05.232309 22541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:05.233359 22541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:05.233438 22541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.233563 22541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:05.233614 22541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.245846 22541 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:32821
I20260809 01:55:05.247215 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:55:05.248636 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:32821 every 8 connection(s)
I20260809 01:55:05.252811 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 22541
I20260809 01:55:05.253381 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:53681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:38279
--local_ip_for_outbound_sockets=127.3.220.2
--tserver_master_addrs=127.3.220.62:41249
--webserver_port=34175
--webserver_interface=127.3.220.2
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.220.20:53681/kms with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:05.273058 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:05.251693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.1:47003)
0809 01:55:05.253711 (+ 2018us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:05.253768 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:05.254609 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:05.256491 (+ 1882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:05.256501 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:05.256885 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:05.257651 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:05.257676 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.259790 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.259796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:05.261432 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:05.261448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.263020 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.263028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:05.263281 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:05.264070 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:05.264098 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:05.266230 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:05.268993 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:05.269000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:05.269016 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:05.269445 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:05.269911 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:05.269914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:05.269916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:05.270111 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:05.270945 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:05.270952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:05.271291 (+ 339us) client_negotiation.cc:769] Sending connection context
0809 01:55:05.271564 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 01:55:05.271919 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1729,"thread_start_us":148,"threads_started":1}
I20260809 01:55:05.274998 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:05.253010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:47003 (local address 127.3.220.62:41249)
0809 01:55:05.253469 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:05.253497 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:05.253912 (+ 415us) server_negotiation.cc:407] Connection header received
0809 01:55:05.255079 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:05.255091 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:05.255546 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:05.255919 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:05.257861 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.259601 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.261634 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.262548 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.262745 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:05.266514 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:05.266525 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:05.266548 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:05.266592 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:05.268840 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:05.269630 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:05.269638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:05.269645 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:05.269806 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:05.270245 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:05.270249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:05.270251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:05.270811 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:05.270963 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:05.273668 (+ 2705us) server_negotiation.cc:299] Negotiation successful
0809 01:55:05.273994 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
I20260809 01:55:05.279057 22780 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:55:05.279263 22780 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:05.279843 22780 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
W20260809 01:55:05.282369 22780 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:41249 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 01:55:05.282495 22780 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:05.282655 22780 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:55:05.283555 22780 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:05.283712 22780 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
W20260809 01:55:05.284457 22780 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 01:55:05.440394 22786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:05.440814 22786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:05.440892 22786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:05.447916 22786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:05.448014 22786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:05.448109 22786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:55:05.457068 22786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.220.20:53681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:38279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=34175
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:05.458456 22786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:05.459630 22786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:05.473188 22796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:05.473394 22794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:05.473706 22793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:05.473958 22786 server_base.cc:1034] running on GCE node
I20260809 01:55:05.474484 22786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:05.475340 22786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:05.476539 22786 hybrid_clock.cc:648] HybridClock initialized: now 1786240505476508 us; error 45 us; skew 500 ppm
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:05.480540 22786 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:55:05.481873 22786 webserver.cc:460] Webserver started at http://127.3.220.2:34175/ using document root <none> and password file <none>
I20260809 01:55:05.482640 22786 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:05.482754 22786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:05.513484 22786 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 01:55:05.516530 22811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.518047 22786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:55:05.518152 22786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "39a35a78693c44cc83cc67b496d4d2ad"
format_stamp: "Formatted at 2026-08-09 01:55:04 on dist-test-slave-gkw7"
server_key: "7d06c063bca541e8eb4b876f903eb970"
server_key_iv: "c3ae444f5dcbe0d71fb97a4e42790ab6"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:05.518877 22786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:05.547003 22786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:05.550951 22786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:05.551159 22786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:05.551826 22786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:05.552703 22786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:05.552749 22786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.552789 22786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:05.552841 22786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.564400 22786 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:38279
I20260809 01:55:05.564420 22973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:38279 every 8 connection(s)
I20260809 01:55:05.565660 22786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:55:05.571173 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 22786
I20260809 01:55:05.571753 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.220.20:53681/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:46691
--local_ip_for_outbound_sockets=127.3.220.3
--tserver_master_addrs=127.3.220.62:41249
--webserver_port=46809
--webserver_interface=127.3.220.3
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.220.20:53681/kms with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:05.583642 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:05.568802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:50827 (local address 127.3.220.62:41249)
0809 01:55:05.568977 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:05.568982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:05.569913 (+ 931us) server_negotiation.cc:407] Connection header received
0809 01:55:05.570948 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:05.570952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:05.571010 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:05.571125 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:05.572535 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.573160 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.574860 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.575223 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.575312 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:05.579009 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:05.579013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:05.579016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:05.579055 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:05.580950 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:05.581664 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:05.581668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:05.581670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:05.581767 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:05.582225 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:05.582229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:05.582232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:05.582478 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:05.582574 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:05.583322 (+ 748us) server_negotiation.cc:299] Negotiation successful
0809 01:55:05.583467 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 01:55:05.585162 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:05.569247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.2:50827)
0809 01:55:05.569734 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:05.569787 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:05.570725 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:05.571337 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:05.571345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:05.571729 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:05.572365 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:05.572381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.573329 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.573333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:05.574706 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:05.574720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.575529 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.575534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:05.575752 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:05.576465 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:05.576482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:05.578803 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:05.581112 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:05.581118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:05.581135 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:05.581506 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:05.581904 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:05.581907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:05.581909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:05.582087 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:05.582655 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:05.582661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:05.582997 (+ 336us) client_negotiation.cc:769] Sending connection context
0809 01:55:05.583824 (+ 827us) client_negotiation.cc:240] Negotiation successful
0809 01:55:05.584103 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":137,"threads_started":1}
I20260809 01:55:05.586597 22977 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:55:05.586879 22977 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:05.587385 22977 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
W20260809 01:55:05.589032 22977 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:41249 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 01:55:05.589180 22977 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:05.589370 22977 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:55:05.590330 22977 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:05.590493 22977 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
W20260809 01:55:05.591362 22977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 01:55:05.716312 22980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:05.716686 22980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:05.716789 22980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:05.721735 22980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:05.721820 22980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:05.721966 22980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:55:05.728317 22980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.220.20:53681/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:46691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=46809
--tserver_master_addrs=127.3.220.62:41249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:05.729487 22980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:05.730726 22980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:05.742992 22993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:05.743917 22990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:05.744990 22991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:05.747483 22980 server_base.cc:1034] running on GCE node
I20260809 01:55:05.748055 22980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:05.748977 22980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:05.750168 22980 hybrid_clock.cc:648] HybridClock initialized: now 1786240505750127 us; error 53 us; skew 500 ppm
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:05.753976 22980 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:55:05.755139 22980 webserver.cc:460] Webserver started at http://127.3.220.3:46809/ using document root <none> and password file <none>
I20260809 01:55:05.755889 22980 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:05.755995 22980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:05.778551 22980 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260809 01:55:05.781674 23001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.783222 22980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:55:05.783376 22980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "ad186e9309774e999b580bcd0466afb5"
format_stamp: "Formatted at 2026-08-09 01:55:04 on dist-test-slave-gkw7"
server_key: "6658df923551c32c19f7d662fcc31eb4"
server_key_iv: "34f03a70a8a55adb1f4483a8988145a5"
server_key_version: "kuduclusterkey@0"
I20260809 01:55:05.784056 22980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:05.818665 22980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:05.822460 22980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:05.822754 22980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:05.823597 22980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:05.824795 22980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:05.824880 22980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.824991 22980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:05.825054 22980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:05.837581 22980 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:46691
I20260809 01:55:05.837677 23179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:46691 every 8 connection(s)
I20260809 01:55:05.838757 22980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:55:05.846096 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 22980
Aug 09 01:55:05 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:05.856026 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:05.840983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:38351 (local address 127.3.220.62:41249)
0809 01:55:05.841176 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:05.841180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:05.842306 (+ 1126us) server_negotiation.cc:407] Connection header received
0809 01:55:05.843570 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:05.843575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:05.843658 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:05.843758 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:05.844962 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.845658 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.848070 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.848327 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.848447 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:05.851603 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:05.851606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:05.851608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:05.851637 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:05.853316 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:05.854041 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:05.854045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:05.854047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:05.854149 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:05.854593 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:05.854597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:05.854599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:05.854831 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:05.854929 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:05.855706 (+ 777us) server_negotiation.cc:299] Negotiation successful
0809 01:55:05.855834 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 01:55:05.857051 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:05.841566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41249 (local address 127.3.220.3:38351)
0809 01:55:05.842137 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:05.842184 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:05.843298 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:05.843933 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:05.843940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:05.844236 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:05.844802 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:05.844817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.845809 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.845813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:05.847918 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:05.847929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.848505 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.848510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:05.848697 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:05.849358 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:05.849376 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:05.851365 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:05.853466 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:05.853472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:05.853492 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:05.853916 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:05.854239 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:05.854245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:05.854247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:05.854462 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:05.854989 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:05.854997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:05.855373 (+ 376us) client_negotiation.cc:769] Sending connection context
0809 01:55:05.855634 (+ 261us) client_negotiation.cc:240] Negotiation successful
0809 01:55:05.855995 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":178,"threads_started":1}
I20260809 01:55:05.858484 23180 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41249
I20260809 01:55:05.858760 23180 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:05.859458 23180 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:55:05.861985 22494 ts_manager.cc:194] Registered new tserver with Master: ad186e9309774e999b580bcd0466afb5 (127.3.220.3:46691)
I20260809 01:55:05.863363 22494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:38351
I20260809 01:55:05.867730 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:05.849735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.3.220.62:41249)
0809 01:55:05.850322 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:05.850333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:05.850349 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:55:05.850402 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:05.850408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:05.850462 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:05.850569 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:05.851963 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.854075 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.857840 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:05.858206 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:05.858311 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:05.860508 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:05.860513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:05.860515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:05.860541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:05.862516 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:05.866253 (+ 3737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:05.866256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:05.866257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:05.866347 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:05.866708 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:05.866712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:05.866714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:05.866906 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:05.867124 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:05.867364 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 01:55:05.867504 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":133,"threads_started":1}
Aug 09 01:55:06 dist-test-slave-gkw7 krb5kdc[21090](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786240506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:06.284694 22780 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:06.285012 22780 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:55:06.285888 22494 ts_manager.cc:194] Registered new tserver with Master: 88dac609f00840f69c390a234fbbd889 (127.3.220.1:32821)
I20260809 01:55:06.286736 22494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:47003
Aug 09 01:55:06 dist-test-slave-gkw7 krb5kdc[21090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for HTTP/127.3.220.20@KRBTEST.COM
I20260809 01:55:06.591706 22977 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:06.592034 22977 heartbeater.cc:511] Master 127.3.220.62:41249 requested a full tablet report, sending...
I20260809 01:55:06.592919 22494 ts_manager.cc:194] Registered new tserver with Master: 39a35a78693c44cc83cc67b496d4d2ad (127.3.220.2:38279)
I20260809 01:55:06.593794 22494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:50827
I20260809 01:55:06.597585 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:55:06.626081 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.600478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.3.220.62:41249)
0809 01:55:06.600758 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:06.600763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:06.611942 (+ 11179us) server_negotiation.cc:407] Connection header received
0809 01:55:06.612162 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:06.612165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:06.612224 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:06.612292 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:06.613350 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.614243 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.615480 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.615833 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.615915 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.621483 (+ 5568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:06.621488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:06.621491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:06.621569 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:06.623889 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:06.624559 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:06.624563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:06.624565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:06.624672 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:06.625026 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:06.625029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:06.625031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:06.625258 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:06.625393 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:06.625751 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 01:55:06.625896 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":118,"threads_started":1}
I20260809 01:55:06.631011 22494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:55:06.636322 22494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:55:06.660305 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.649056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:32821 (local address 127.0.0.1:36906)
0809 01:55:06.650969 (+ 1913us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:06.651020 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:06.651148 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:06.652110 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:06.652117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:06.652146 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:06.652302 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.652309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.656356 (+ 4047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.656359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.657549 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.657559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.659674 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.659678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.659811 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.659860 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 01:55:06.659989 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 01:55:06.660132 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1725,"thread_start_us":129,"threads_started":1}
I20260809 01:55:06.660708 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.649173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.3.220.1:32821)
0809 01:55:06.651596 (+ 2423us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:06.651602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:06.651626 (+ 24us) server_negotiation.cc:407] Connection header received
0809 01:55:06.651718 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:06.651723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:06.651933 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:06.652089 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:06.652476 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.654589 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.658430 (+ 3841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.659304 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.659424 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.659584 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:06.660431 (+ 847us) server_negotiation.cc:299] Negotiation successful
0809 01:55:06.660549 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2319,"thread_start_us":128,"threads_started":1}
I20260809 01:55:06.661465 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.650326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.3.220.3:46691)
0809 01:55:06.651425 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:06.651432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:06.652462 (+ 1030us) server_negotiation.cc:407] Connection header received
0809 01:55:06.652663 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:06.652668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:06.652870 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:06.652978 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:06.653429 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.654612 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.656322 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.657236 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.657350 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.657512 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:06.661203 (+ 3691us) server_negotiation.cc:299] Negotiation successful
0809 01:55:06.661323 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":97,"threads_started":1}
I20260809 01:55:06.661943 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.649429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:46691 (local address 127.0.0.1:50136)
0809 01:55:06.652365 (+ 2936us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:06.652379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:06.652529 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:06.653191 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:06.653194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:06.653212 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:06.653309 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.653315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.654767 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.654770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.656170 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.656180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.660918 (+ 4738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.660922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.661027 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.661051 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 01:55:06.661701 (+ 650us) client_negotiation.cc:240] Negotiation successful
0809 01:55:06.661787 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2869,"thread_start_us":121,"threads_started":1}
I20260809 01:55:06.664623 22678 tablet_service.cc:1467] Processing CreateTablet for tablet 71d0117c76bb4ad88f1c29ce80836510 (DEFAULT_TABLE table=test-table [id=8fe8548b3ef743d99d94b3219f6cd005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:06.664921 23064 tablet_service.cc:1467] Processing CreateTablet for tablet 71d0117c76bb4ad88f1c29ce80836510 (DEFAULT_TABLE table=test-table [id=8fe8548b3ef743d99d94b3219f6cd005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:06.666035 23064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d0117c76bb4ad88f1c29ce80836510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:06.666043 22678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d0117c76bb4ad88f1c29ce80836510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:06.678983 23220 tablet_bootstrap.cc:492] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: Bootstrap starting.
I20260809 01:55:06.679150 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.649938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38279 (local address 127.0.0.1:37148)
0809 01:55:06.657815 (+ 7877us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:06.657829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:06.657916 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:06.658643 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:06.658647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:06.658664 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:06.658773 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.658779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.660182 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.660186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.677144 (+ 16958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.677160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.678709 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.678713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.678824 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.678848 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 01:55:06.678915 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 01:55:06.678997 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7809,"thread_start_us":86,"threads_started":1}
I20260809 01:55:06.679589 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.650153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.3.220.2:38279)
0809 01:55:06.650555 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:06.650563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:06.658021 (+ 7458us) server_negotiation.cc:407] Connection header received
0809 01:55:06.658110 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:06.658116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:06.658334 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:06.658481 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:06.658902 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.660047 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.677334 (+ 17287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.678359 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.678448 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.678590 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:06.679351 (+ 761us) server_negotiation.cc:299] Negotiation successful
0809 01:55:06.679453 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":105,"threads_started":1}
I20260809 01:55:06.680157 23221 tablet_bootstrap.cc:492] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889: Bootstrap starting.
I20260809 01:55:06.682711 23221 tablet_bootstrap.cc:654] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:06.683712 23221 log.cc:824] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:06.684221 23220 tablet_bootstrap.cc:654] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:06.684607 22869 tablet_service.cc:1467] Processing CreateTablet for tablet 71d0117c76bb4ad88f1c29ce80836510 (DEFAULT_TABLE table=test-table [id=8fe8548b3ef743d99d94b3219f6cd005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:06.685195 23220 log.cc:824] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:06.685981 22869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d0117c76bb4ad88f1c29ce80836510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:06.687683 23220 tablet_bootstrap.cc:492] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: No bootstrap required, opened a new log
I20260809 01:55:06.687913 23220 ts_tablet_manager.cc:1397] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260809 01:55:06.693050 23220 raft_consensus.cc:348] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.693354 23220 raft_consensus.cc:374] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:06.693418 23220 raft_consensus.cc:729] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad186e9309774e999b580bcd0466afb5, State: Initialized, Role: FOLLOWER
I20260809 01:55:06.695204 23220 consensus_queue.cc:260] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.696447 23220 ts_tablet_manager.cc:1428] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260809 01:55:06.696983 23180 heartbeater.cc:503] Master 127.3.220.62:41249 was elected leader, sending a full tablet report...
I20260809 01:55:06.697504 23228 tablet_bootstrap.cc:492] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Bootstrap starting.
I20260809 01:55:06.700418 23228 tablet_bootstrap.cc:654] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:06.701404 23228 log.cc:824] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:06.704416 23228 tablet_bootstrap.cc:492] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: No bootstrap required, opened a new log
I20260809 01:55:06.704677 23228 ts_tablet_manager.cc:1397] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260809 01:55:06.707623 23221 tablet_bootstrap.cc:492] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889: No bootstrap required, opened a new log
I20260809 01:55:06.707849 23221 ts_tablet_manager.cc:1397] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20260809 01:55:06.709120 23228 raft_consensus.cc:348] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.709414 23228 raft_consensus.cc:374] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:06.709477 23228 raft_consensus.cc:729] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a35a78693c44cc83cc67b496d4d2ad, State: Initialized, Role: FOLLOWER
I20260809 01:55:06.710057 23228 consensus_queue.cc:260] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.711236 23228 ts_tablet_manager.cc:1428] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260809 01:55:06.711647 22977 heartbeater.cc:503] Master 127.3.220.62:41249 was elected leader, sending a full tablet report...
I20260809 01:55:06.715065 23221 raft_consensus.cc:348] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.715342 23221 raft_consensus.cc:374] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:06.715417 23221 raft_consensus.cc:729] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88dac609f00840f69c390a234fbbd889, State: Initialized, Role: FOLLOWER
I20260809 01:55:06.715909 23221 consensus_queue.cc:260] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.716861 23221 ts_tablet_manager.cc:1428] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260809 01:55:06.717268 22780 heartbeater.cc:503] Master 127.3.220.62:41249 was elected leader, sending a full tablet report...
W20260809 01:55:06.750476 22781 tablet.cc:2369] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:55:06.752169 23232 raft_consensus.cc:482] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:06.752372 23232 raft_consensus.cc:504] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.753569 23232 leader_election.cc:288] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad186e9309774e999b580bcd0466afb5 (127.3.220.3:46691), 39a35a78693c44cc83cc67b496d4d2ad (127.3.220.2:38279)
I20260809 01:55:06.759050 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.753976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:50701 (local address 127.3.220.3:46691)
0809 01:55:06.754111 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:06.754118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:06.754380 (+ 262us) server_negotiation.cc:407] Connection header received
0809 01:55:06.754606 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:06.754610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:06.754686 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:06.754741 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:06.755231 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.755997 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.757650 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.758288 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.758369 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.758441 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:06.758808 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 01:55:06.758895 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:55:06.759050 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.754623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:54227 (local address 127.3.220.2:38279)
0809 01:55:06.754755 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:06.754759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:06.755023 (+ 264us) server_negotiation.cc:407] Connection header received
0809 01:55:06.755239 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:06.755243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:06.755295 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:06.755344 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:06.755712 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.756420 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.757825 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.758413 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.758489 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.758546 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:06.758857 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 01:55:06.758928 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:55:06.759508 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.754258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38279 (local address 127.3.220.1:54227)
0809 01:55:06.754942 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:06.754954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:06.755095 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:06.755478 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:06.755481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:06.755497 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:06.755595 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.755600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.756557 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.756560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.757701 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.757711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.758626 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.758629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.758720 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.758747 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:55:06.759269 (+ 522us) client_negotiation.cc:240] Negotiation successful
0809 01:55:06.759359 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":95,"threads_started":1}
I20260809 01:55:06.760195 22877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d0117c76bb4ad88f1c29ce80836510" candidate_uuid: "88dac609f00840f69c390a234fbbd889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a35a78693c44cc83cc67b496d4d2ad" is_pre_election: true
I20260809 01:55:06.760548 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.753800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:46691 (local address 127.3.220.1:50701)
0809 01:55:06.754277 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:06.754291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:06.754471 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:06.754902 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:06.754906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:06.754922 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:06.755097 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.755104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.756131 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.756134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.757463 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:06.757474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.758538 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.758542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:06.758645 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.758681 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 01:55:06.759808 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0809 01:55:06.759880 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":9,"thread_start_us":142,"threads_started":1}
I20260809 01:55:06.760535 22877 raft_consensus.cc:2454] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88dac609f00840f69c390a234fbbd889 in term 0.
I20260809 01:55:06.761389 22620 leader_election.cc:302] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a35a78693c44cc83cc67b496d4d2ad, 88dac609f00840f69c390a234fbbd889; no voters:
I20260809 01:55:06.761441 23120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d0117c76bb4ad88f1c29ce80836510" candidate_uuid: "88dac609f00840f69c390a234fbbd889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad186e9309774e999b580bcd0466afb5" is_pre_election: true
I20260809 01:55:06.761744 23120 raft_consensus.cc:2454] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88dac609f00840f69c390a234fbbd889 in term 0.
I20260809 01:55:06.761859 23232 raft_consensus.cc:2781] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:55:06.761938 23232 raft_consensus.cc:482] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:55:06.761983 23232 raft_consensus.cc:3037] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:06.763351 23232 raft_consensus.cc:504] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.763736 23232 leader_election.cc:288] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [CANDIDATE]: Term 1 election: Requested vote from peers ad186e9309774e999b580bcd0466afb5 (127.3.220.3:46691), 39a35a78693c44cc83cc67b496d4d2ad (127.3.220.2:38279)
I20260809 01:55:06.764091 23120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d0117c76bb4ad88f1c29ce80836510" candidate_uuid: "88dac609f00840f69c390a234fbbd889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad186e9309774e999b580bcd0466afb5"
I20260809 01:55:06.764093 22877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d0117c76bb4ad88f1c29ce80836510" candidate_uuid: "88dac609f00840f69c390a234fbbd889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a35a78693c44cc83cc67b496d4d2ad"
I20260809 01:55:06.764212 23120 raft_consensus.cc:3037] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:06.764216 22877 raft_consensus.cc:3037] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:06.765515 22877 raft_consensus.cc:2454] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88dac609f00840f69c390a234fbbd889 in term 1.
I20260809 01:55:06.765515 23120 raft_consensus.cc:2454] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88dac609f00840f69c390a234fbbd889 in term 1.
I20260809 01:55:06.765933 22620 leader_election.cc:302] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88dac609f00840f69c390a234fbbd889, ad186e9309774e999b580bcd0466afb5; no voters:
I20260809 01:55:06.766211 23232 raft_consensus.cc:2781] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:06.766477 23232 raft_consensus.cc:686] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [term 1 LEADER]: Becoming Leader. State: Replica: 88dac609f00840f69c390a234fbbd889, State: Running, Role: LEADER
I20260809 01:55:06.766830 23232 consensus_queue.cc:237] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } }
I20260809 01:55:06.770419 22493 catalog_manager.cc:5495] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88dac609f00840f69c390a234fbbd889 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "88dac609f00840f69c390a234fbbd889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dac609f00840f69c390a234fbbd889" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 32821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 } health_report { overall_health: UNKNOWN } } }
I20260809 01:55:06.797501 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.792160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.3.220.1:32821)
0809 01:55:06.792282 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:06.792288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:06.792338 (+ 50us) server_negotiation.cc:407] Connection header received
0809 01:55:06.792549 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:06.792552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:06.792616 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:06.792738 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:55:06.793029 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.793789 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.795093 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:06.795454 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:06.795553 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:06.796031 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:55:06.796654 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:55:06.796777 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:06.797260 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 01:55:06.797329 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 01:55:06.804365 22877 raft_consensus.cc:1261] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 1 FOLLOWER]: Refusing update from remote peer 88dac609f00840f69c390a234fbbd889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:06.804527 23120 raft_consensus.cc:1261] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 1 FOLLOWER]: Refusing update from remote peer 88dac609f00840f69c390a234fbbd889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:06.805241 23232 consensus_queue.cc:1035] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a35a78693c44cc83cc67b496d4d2ad" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:06.805610 23235 consensus_queue.cc:1035] T 71d0117c76bb4ad88f1c29ce80836510 P 88dac609f00840f69c390a234fbbd889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad186e9309774e999b580bcd0466afb5" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 01:55:06.818367 22978 tablet.cc:2369] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:55:06.818357 23241 mvcc.cc:205] Tried to move back new op lower bound from 7316441115862511616 to 7316441115720044544. Current Snapshot: MvccSnapshot[applied={T|T < 7316441115862511616}]
I20260809 01:55:06.824067 23243 mvcc.cc:205] Tried to move back new op lower bound from 7316441115862511616 to 7316441115720044544. Current Snapshot: MvccSnapshot[applied={T|T < 7316441115862511616}]
I20260809 01:55:06.824354 23246 mvcc.cc:205] Tried to move back new op lower bound from 7316441115862511616 to 7316441115720044544. Current Snapshot: MvccSnapshot[applied={T|T < 7316441115862511616}]
I20260809 01:55:06.834964 22493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:06.835155 22493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:06.838004 22493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e501253e2dfc4235bff3972fccaa8c0d: Sending DeleteTablet for 3 replicas of tablet 71d0117c76bb4ad88f1c29ce80836510
I20260809 01:55:06.838965 22678 tablet_service.cc:1514] Processing DeleteTablet for tablet 71d0117c76bb4ad88f1c29ce80836510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:06 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:36906
I20260809 01:55:06.840389 23064 tablet_service.cc:1514] Processing DeleteTablet for tablet 71d0117c76bb4ad88f1c29ce80836510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:06 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:50136
I20260809 01:55:06.840895 22869 tablet_service.cc:1514] Processing DeleteTablet for tablet 71d0117c76bb4ad88f1c29ce80836510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:06 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:37148
I20260809 01:55:06.842072 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 22541
I20260809 01:55:06.853015 23255 tablet_replica.cc:330] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: stopping tablet replica
I20260809 01:55:06.853507 23255 raft_consensus.cc:2229] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:06.853955 23255 raft_consensus.cc:2258] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:06.854358 23257 tablet_replica.cc:330] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: stopping tablet replica
I20260809 01:55:06.854687 23257 raft_consensus.cc:2229] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:06.855000 23257 raft_consensus.cc:2258] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:06.856671 23255 ts_tablet_manager.cc:1905] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:06.859634 23255 ts_tablet_manager.cc:1918] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:06.859803 23255 log.cc:1196] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/71d0117c76bb4ad88f1c29ce80836510
I20260809 01:55:06.860224 23255 ts_tablet_manager.cc:1939] T 71d0117c76bb4ad88f1c29ce80836510 P ad186e9309774e999b580bcd0466afb5: Deleting consensus metadata
I20260809 01:55:06.861694 22478 catalog_manager.cc:4841] TS ad186e9309774e999b580bcd0466afb5 (127.3.220.3:46691): tablet 71d0117c76bb4ad88f1c29ce80836510 (table test-table [id=8fe8548b3ef743d99d94b3219f6cd005]) successfully deleted
I20260809 01:55:06.866302 23257 ts_tablet_manager.cc:1905] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:06.869167 23257 ts_tablet_manager.cc:1918] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:06.869413 23257 log.cc:1196] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/71d0117c76bb4ad88f1c29ce80836510
I20260809 01:55:06.869972 23257 ts_tablet_manager.cc:1939] T 71d0117c76bb4ad88f1c29ce80836510 P 39a35a78693c44cc83cc67b496d4d2ad: Deleting consensus metadata
I20260809 01:55:06.871838 22478 catalog_manager.cc:4841] TS 39a35a78693c44cc83cc67b496d4d2ad (127.3.220.2:38279): tablet 71d0117c76bb4ad88f1c29ce80836510 (table test-table [id=8fe8548b3ef743d99d94b3219f6cd005]) successfully deleted
W20260809 01:55:06.872874 22471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:32821 (error 108)
I20260809 01:55:06.873850 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 22786
I20260809 01:55:06.876437 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:06.875081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:32821 (local address 127.0.0.1:36926)
0809 01:55:06.876244 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:06.876329 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1057}
W20260809 01:55:06.876751 22471 catalog_manager.cc:4584] TS 88dac609f00840f69c390a234fbbd889 (127.3.220.1:32821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71d0117c76bb4ad88f1c29ce80836510: Network error: Client connection negotiation failed: client connection to 127.3.220.1:32821: connect: Connection refused (error 111)
I20260809 01:55:06.886411 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 22980
I20260809 01:55:06.899098 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 22447
I20260809 01:55:06.911250 3952 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 01:55:07.257695 3952 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 01:55:07.257814 3952 mini_ranger.cc:66] Stopping Ranger...
I20260809 01:55:07.603278 3952 mini_ranger.cc:68] Stopped Ranger
2026-08-09 01:55:07.603 UTC [21132] LOG: received smart shutdown request
2026-08-09 01:55:07.607 UTC [21132] LOG: background worker "logical replication launcher" (PID 21146) exited with exit code 1
2026-08-09 01:55:07.608 UTC [21139] LOG: shutting down
2026-08-09 01:55:07.783 UTC [21132] LOG: database system is shut down
2026-08-09T01:55:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52089 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:55:07 dist-test-slave-gkw7 krb5kdc[23271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:55:07 dist-test-slave-gkw7 krb5kdc[23271](info): set up 2 sockets
Aug 09 01:55:07 dist-test-slave-gkw7 krb5kdc[23271](info): commencing operation
krb5kdc: starting...
W20260809 01:55:09.146562 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:55:09.281183 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35817
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:35817,127.3.220.61:37041,127.3.220.60:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:09.395193 23291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:09.395536 23291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:09.395576 23291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:09.399698 23291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:09.399755 23291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:09.399775 23291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:09.399791 23291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:09.399843 23291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:09.406901 23291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:35817,127.3.220.61:37041,127.3.220.60:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:09.408360 23291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:09.409828 23291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:09.419927 23297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:09.419955 23296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:09.420159 23301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:09.420415 23291 server_base.cc:1034] running on GCE node
I20260809 01:55:09.420958 23291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:09.422165 23291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:09.423353 23291 hybrid_clock.cc:648] HybridClock initialized: now 1786240509423333 us; error 31 us; skew 500 ppm
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:09.426172 23291 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:09.427075 23291 webserver.cc:460] Webserver started at http://127.3.220.62:42585/ using document root <none> and password file <none>
I20260809 01:55:09.427668 23291 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:09.427763 23291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:09.428009 23291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:09.430197 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "d479a314af154f7895d715ca614156ae"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "2a6ceb9bba0629aa75dc9f1e033a4261"
server_key_iv: "72fff0f3c11b82311edc378026378360"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.430778 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "d479a314af154f7895d715ca614156ae"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "2a6ceb9bba0629aa75dc9f1e033a4261"
server_key_iv: "72fff0f3c11b82311edc378026378360"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.434481 23291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 01:55:09.436873 23315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:09.438189 23291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:55:09.438336 23291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "d479a314af154f7895d715ca614156ae"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "2a6ceb9bba0629aa75dc9f1e033a4261"
server_key_iv: "72fff0f3c11b82311edc378026378360"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.438455 23291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:09.452483 23291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:09.455363 23291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:09.455569 23291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:09.463043 23291 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:35817
I20260809 01:55:09.463054 23396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:35817 every 8 connection(s)
I20260809 01:55:09.464028 23291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:09.466897 23399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:09.468369 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 23291
I20260809 01:55:09.468492 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:55:09.468833 3952 external_mini_cluster.cc:1428] Setting key 0046c1b1902c03805ff6b5342910684b
I20260809 01:55:09.471900 23399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.220.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.61@KRBTEST.COM" created.
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:09.487195 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.474341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35817 (local address 127.0.0.1:48070)
0809 01:55:09.474877 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.474918 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.475785 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.476459 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.476464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.476884 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.477123 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.477136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.478424 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.478427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.479164 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.479173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.479808 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.479812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.479978 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.480721 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.480738 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.482232 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.484157 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.484166 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.484176 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.484404 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.484730 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.484734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.484736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.484857 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.485450 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.485456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.485628 (+ 172us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.485816 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.486200 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":124,"thread_start_us":165,"threads_started":1}
I20260809 01:55:09.487205 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.475397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.3.220.62:35817)
0809 01:55:09.475722 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.475733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.475778 (+ 45us) server_negotiation.cc:407] Connection header received
0809 01:55:09.476092 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.476096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.476268 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.476460 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.477270 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.478269 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.479281 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.479654 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.479903 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.482391 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.482397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.482407 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.482427 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.484027 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.484519 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.484525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.484526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.484635 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.484959 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.484962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.484963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.485335 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.485479 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.485878 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.486149 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
I20260809 01:55:09.489780 23399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:55:09.490806 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.490276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37736)
0809 01:55:09.490577 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.490693 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 01:55:09.491089 23321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
I20260809 01:55:09.492019 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.491812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37740)
0809 01:55:09.491924 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.491968 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 01:55:09.492170 23399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37041: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab.
Entry for principal kudu/127.3.220.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab.
I20260809 01:55:09.496778 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.61:37041
--webserver_interface=127.3.220.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab
--principal=kudu/127.3.220.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:35817,127.3.220.61:37041,127.3.220.60:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:09.541466 23399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } attempt: 1
I20260809 01:55:09.542663 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.542290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37756)
0809 01:55:09.542480 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.542559 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 01:55:09.543553 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.543287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37766)
0809 01:55:09.543432 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.543487 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260809 01:55:09.543699 23399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37041: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
I20260809 01:55:09.611901 23399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } attempt: 2
I20260809 01:55:09.612869 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.612485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37772)
0809 01:55:09.612689 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.612781 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260809 01:55:09.613058 23410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:09.613445 23410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:09.613523 23410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 01:55:09.613835 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.613584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37780)
0809 01:55:09.613726 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.613777 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 01:55:09.614058 23399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37041: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37041: connect: Connection refused (error 111)
W20260809 01:55:09.617830 23410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:09.617884 23410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:09.617905 23410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:09.617950 23410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:09.617983 23410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:09.623471 23410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:35817,127.3.220.61:37041,127.3.220.60:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab
--principal=kudu/127.3.220.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.61:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.220.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:09.624565 23410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:09.625711 23410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:09.634577 23419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:09.634570 23420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:09.634582 23422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:09.634866 23410 server_base.cc:1034] running on GCE node
I20260809 01:55:09.635497 23410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:09.636658 23410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:09.637849 23410 hybrid_clock.cc:648] HybridClock initialized: now 1786240509637830 us; error 30 us; skew 500 ppm
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:09.640720 23410 init.cc:377] Logged in from keytab as kudu/127.3.220.61@KRBTEST.COM (short username kudu)
I20260809 01:55:09.641825 23410 webserver.cc:460] Webserver started at http://127.3.220.61:44641/ using document root <none> and password file <none>
I20260809 01:55:09.642454 23410 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:09.642550 23410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:09.642766 23410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:09.644841 23410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/data/instance:
uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "e30ce7ccfd8a1a4d6a89ad5669e5a857"
server_key_iv: "ffb93d9a26cf840adc1340e04701f5cb"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.645434 23410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/wal/instance:
uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "e30ce7ccfd8a1a4d6a89ad5669e5a857"
server_key_iv: "ffb93d9a26cf840adc1340e04701f5cb"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.649179 23410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:55:09.651877 23435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:09.653244 23410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:55:09.653409 23410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/wal
uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "e30ce7ccfd8a1a4d6a89ad5669e5a857"
server_key_iv: "ffb93d9a26cf840adc1340e04701f5cb"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.653527 23410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:09.676332 23410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:09.680034 23410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:09.680287 23410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:09.687469 23410 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.61:37041
I20260809 01:55:09.687528 23516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.61:37041 every 8 connection(s)
I20260809 01:55:09.688498 23410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
I20260809 01:55:09.691368 23517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:09.694589 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 23410
I20260809 01:55:09.694729 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-1/wal/instance
I20260809 01:55:09.695061 3952 external_mini_cluster.cc:1428] Setting key c926cde6d7a0306740a3877c43cf827d
I20260809 01:55:09.695501 23517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:55:09.705257 23399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } attempt: 3
WARNING: no policy specified for kudu/127.3.220.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.60@KRBTEST.COM" created.
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:55:09.715572 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.698058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.3.220.62:35817)
0809 01:55:09.698240 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.698244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.699399 (+ 1155us) server_negotiation.cc:407] Connection header received
0809 01:55:09.700978 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.700985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.701069 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.701192 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.702625 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.703658 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.704937 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.705450 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.705659 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.710553 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.710558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.710562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.710600 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.713134 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.713812 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.713816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.713818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.713909 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.714312 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.714316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.714318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.714523 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.714645 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.715240 (+ 595us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.715377 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 01:55:09.716648 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.698614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35817 (local address 127.0.0.1:48082)
0809 01:55:09.699220 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.699269 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.700695 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.701391 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.701398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.701896 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.702443 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.702458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.703825 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.703828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.704789 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.704798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.705712 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.705718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.705941 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.706819 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.706842 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.710379 (+ 3537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.713269 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.713282 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.713295 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.713696 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.714017 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.714021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.714023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.714194 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.714646 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.714655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.714954 (+ 299us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.715170 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.715527 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":171,"threads_started":1}
I20260809 01:55:09.717905 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.705804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37786)
0809 01:55:09.705961 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.705982 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.706079 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.706972 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.706975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.707197 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.707311 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.707317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.708465 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.708469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.709156 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.709187 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.709982 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.709987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.710117 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.710917 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.710934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.712689 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.715534 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.715540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.715546 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.715909 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.716281 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.716285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.716287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.716387 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.717125 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.717132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.717269 (+ 137us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.717403 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.717630 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 01:55:09.718356 23517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:55:09.718786 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.706136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.3.220.61:37041)
0809 01:55:09.706540 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.706554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.706577 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:55:09.706615 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.706619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.706798 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.706989 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.707441 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.708260 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.709335 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.709831 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.709982 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.712892 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.712904 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.712924 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.712953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.715386 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.716046 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.716053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.716055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.716150 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.716517 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.716525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.716529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.716906 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.717062 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.718286 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.718478 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":141,"threads_started":1}
I20260809 01:55:09.720718 23399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:55:09.721446 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.721194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57896)
0809 01:55:09.721315 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.721373 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 01:55:09.722231 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.721977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57910)
0809 01:55:09.722122 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.722179 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260809 01:55:09.722397 23399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:38395: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:55:09.728391 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.718930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37790)
0809 01:55:09.719313 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.719328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.719421 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.719707 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.719710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.719892 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.720013 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.720020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.721278 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.721284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.722243 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.722254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.722879 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.722882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.723010 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.723547 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.723559 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.724939 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.726693 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.726696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.726697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.727024 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.727288 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.727291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.727297 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.727406 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.727762 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.727764 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.727883 (+ 119us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.728046 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.728234 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":28}
I20260809 01:55:09.728410 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.718966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.3.220.61:37041)
0809 01:55:09.719095 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.719101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.719435 (+ 334us) server_negotiation.cc:407] Connection header received
0809 01:55:09.719492 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.719495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.719550 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.719675 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.720153 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.721120 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.722429 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.722739 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.722882 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.725063 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.725066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.725068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.725089 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.726564 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.727128 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.727131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.727132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.727203 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.727511 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.727513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.727514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.727674 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.727749 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.728097 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.728283 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":50}
I20260809 01:55:09.729286 23517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:55:09.730028 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.729717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57926)
0809 01:55:09.729885 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.729972 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 01:55:09.730813 23437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab.
Entry for principal kudu/127.3.220.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab.
I20260809 01:55:09.731959 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.731724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57930)
0809 01:55:09.731856 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.731912 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 01:55:09.732110 23517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:38395: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
I20260809 01:55:09.732515 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.60:38395
--webserver_interface=127.3.220.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab
--principal=kudu/127.3.220.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:35817,127.3.220.61:37041,127.3.220.60:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:09.753633 23399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } attempt: 1
I20260809 01:55:09.754554 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.754202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57936)
0809 01:55:09.754392 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.754471 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 01:55:09.755398 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.755148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57950)
0809 01:55:09.755277 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.755338 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 01:55:09.755548 23399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:38395: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
I20260809 01:55:09.781401 23517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } attempt: 1
I20260809 01:55:09.782451 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.782108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57962)
0809 01:55:09.782291 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.782365 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 01:55:09.783293 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.783037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57970)
0809 01:55:09.783160 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.783224 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 01:55:09.783421 23517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:38395: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
I20260809 01:55:09.830760 23399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } attempt: 2
I20260809 01:55:09.831689 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.831350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57976)
0809 01:55:09.831521 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.831590 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 01:55:09.832578 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.832328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57990)
0809 01:55:09.832461 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.832520 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 01:55:09.832732 23399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:38395: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
W20260809 01:55:09.848420 23526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:09.848722 23526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:09.848858 23526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 01:55:09.851598 23517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } attempt: 2
I20260809 01:55:09.852487 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.852159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:57994)
0809 01:55:09.852339 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.852406 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 01:55:09.853261 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.853030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:58002)
0809 01:55:09.853152 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.853209 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 01:55:09.853267 23526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:09.853353 23526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:09.853389 23526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:09.853415 23526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:09.853397 23517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:38395: Network error: Client connection negotiation failed: client connection to 127.3.220.60:38395: connect: Connection refused (error 111)
W20260809 01:55:09.853452 23526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:09.859107 23526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:35817,127.3.220.61:37041,127.3.220.60:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab
--principal=kudu/127.3.220.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.60:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.220.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:09.860203 23526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:09.861241 23526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:09.869776 23532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:09.869771 23536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:09.869908 23526 server_base.cc:1034] running on GCE node
W20260809 01:55:09.869767 23533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:09.870646 23526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:09.871919 23526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:09.873135 23526 hybrid_clock.cc:648] HybridClock initialized: now 1786240509873110 us; error 42 us; skew 500 ppm
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:09.876497 23526 init.cc:377] Logged in from keytab as kudu/127.3.220.60@KRBTEST.COM (short username kudu)
I20260809 01:55:09.877502 23526 webserver.cc:460] Webserver started at http://127.3.220.60:45499/ using document root <none> and password file <none>
I20260809 01:55:09.878216 23526 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:09.878328 23526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:09.878613 23526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:09.880914 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/data/instance:
uuid: "4efb47143a4448fc97291cceac7fcd92"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "3ffd9a1ce3e5717c307fed15d9fa8d38"
server_key_iv: "4b79a2c27a5d58ed3c72f62de82beb77"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.881680 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/wal/instance:
uuid: "4efb47143a4448fc97291cceac7fcd92"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "3ffd9a1ce3e5717c307fed15d9fa8d38"
server_key_iv: "4b79a2c27a5d58ed3c72f62de82beb77"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.885895 23526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:55:09.888367 23545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:09.889750 23526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:55:09.889891 23526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/wal
uuid: "4efb47143a4448fc97291cceac7fcd92"
format_stamp: "Formatted at 2026-08-09 01:55:09 on dist-test-slave-gkw7"
server_key: "3ffd9a1ce3e5717c307fed15d9fa8d38"
server_key_iv: "4b79a2c27a5d58ed3c72f62de82beb77"
server_key_version: "encryptionkey@0"
I20260809 01:55:09.890004 23526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:09.914620 23526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:09.918272 23526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:09.918500 23526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:09.926247 23526 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.60:38395
I20260809 01:55:09.926270 23620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.60:38395 every 8 connection(s)
I20260809 01:55:09.927263 23526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
I20260809 01:55:09.929867 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 23526
I20260809 01:55:09.930023 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786240338176169-3952-0/minicluster-data/master-2/wal/instance
I20260809 01:55:09.930390 23621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:09.930403 3952 external_mini_cluster.cc:1428] Setting key 15d7b036c9cf5b561a55c73ff3d0a712
I20260809 01:55:09.931914 23399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } attempt: 3
I20260809 01:55:09.935923 23621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
I20260809 01:55:09.944846 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.932424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:58016)
0809 01:55:09.932588 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.932604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.932710 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.934050 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.934053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.934277 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.934406 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.934413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.936003 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.936008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.936851 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.936863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.937687 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.937691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.937787 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.938410 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.938425 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.940083 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.942809 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.942813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.942816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.943100 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.943453 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.943456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.943458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.943555 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.944205 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.944208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.944355 (+ 147us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.944482 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.944646 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:09.944993 23517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } attempt: 3
I20260809 01:55:09.946076 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.932858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.3.220.60:38395)
0809 01:55:09.933149 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.933156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.933175 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:55:09.933336 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.933340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.933756 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.934526 (+ 770us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.934654 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.935846 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.936984 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.937506 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.938030 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.940297 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.940302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.940312 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.940336 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.942646 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.943219 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.943223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.943228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.943339 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.943659 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.943663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.943666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.944057 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.944213 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.944645 (+ 432us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.944957 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":234,"spinlock_wait_cycles":14976,"thread_start_us":111,"threads_started":1}
I20260809 01:55:09.946255 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.931882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.3.220.60:38395)
0809 01:55:09.932404 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.932424 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.932469 (+ 45us) server_negotiation.cc:407] Connection header received
0809 01:55:09.933318 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.933338 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.933755 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.934090 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.934928 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.935807 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.937030 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.937502 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.937712 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.941305 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.941309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.941311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.941336 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.943800 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.944414 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.944417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.944419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.944554 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.944888 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.944891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.944896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.945080 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.945910 (+ 830us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.945986 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.946097 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":362,"spinlock_wait_cycles":12544,"thread_start_us":144,"threads_started":1}
I20260809 01:55:09.949719 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.938530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.3.220.62:35817)
0809 01:55:09.938663 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.938667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.938681 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:55:09.938718 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.938727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.938760 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.938849 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.939515 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.940080 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.941789 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.942139 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.942270 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.945831 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.945835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.945838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.945862 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.947690 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.948221 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.948225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.948226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.948310 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.948613 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.948616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.948618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.948903 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.948983 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.949408 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.949572 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 01:55:09.950651 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.938021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35817 (local address 127.0.0.1:48090)
0809 01:55:09.938415 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.938479 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.938593 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.938902 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.938909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.939271 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.939388 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.939393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.940603 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.940606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.941638 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.941651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.942272 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.942276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.942401 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.942927 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.942965 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.945678 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.947812 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.947821 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.947827 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.948115 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.948422 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.948425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.948427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.948514 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.949070 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.949076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.949183 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.950209 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.950424 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
I20260809 01:55:09.951004 23399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae: Bootstrap starting.
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
I20260809 01:55:09.951726 23621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:55:09.955294 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.945586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:58022)
0809 01:55:09.945853 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.945870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.945943 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.946745 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.946750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.946951 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.947076 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.947084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.948108 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.948112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.948922 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.948935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.949550 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.949554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.949641 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.950124 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.950139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.951528 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.953627 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.953630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.953632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.953909 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.954226 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.954230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.954232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.954323 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.954824 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.954827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.954922 (+ 95us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.955000 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.955144 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260809 01:55:09.955551 23399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae: Neither blocks nor log segments found. Creating new log.
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:09.955966 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.946340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.3.220.60:38395)
0809 01:55:09.946524 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.946528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.946539 (+ 11us) server_negotiation.cc:407] Connection header received
0809 01:55:09.946570 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.946572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.946609 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.946700 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.947236 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.947975 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.949062 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.949400 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.949511 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.951692 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.951695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.951698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.951721 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.953469 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.954039 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.954042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.954044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.954122 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.954429 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.954431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.954433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.954706 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.955587 (+ 881us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.955676 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.955795 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":129}
I20260809 01:55:09.956792 23399 log.cc:824] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:55:09.959097 23399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae: No bootstrap required, opened a new log
I20260809 01:55:09.959277 23517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0: Bootstrap starting.
I20260809 01:55:09.961949 23517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:09.962893 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.952226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37041 (local address 127.0.0.1:37798)
0809 01:55:09.952351 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.952366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.952435 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.952860 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.952863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.953064 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.953172 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.953178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.954509 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.954515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.955436 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.955449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.956212 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.956217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.956337 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.956911 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.956927 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.958975 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.961039 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.961042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.961044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.961301 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.961703 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.961706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.961708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.961800 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.962365 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.962368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.962467 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.962580 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.962739 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 01:55:09.962922 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.952505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.3.220.61:37041)
0809 01:55:09.952641 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.952647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.952662 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:55:09.952703 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.952706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.952750 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.952839 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.953301 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.954352 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.955584 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.955969 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.956046 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.959130 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.959134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.959137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.959166 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.960911 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.961415 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.961419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.961421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.961574 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.961926 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.961930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.961932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.962245 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.962364 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.962654 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.962782 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:55:09.962883 23517 log.cc:824] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:09.963508 23399 raft_consensus.cc:348] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:09.963774 23399 raft_consensus.cc:374] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:09.963765 23621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:55:09.963841 23399 raft_consensus.cc:729] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d479a314af154f7895d715ca614156ae, State: Initialized, Role: FOLLOWER
I20260809 01:55:09.964358 23399 consensus_queue.cc:260] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:09.964859 23633 sys_catalog.cc:455] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:09.965058 23633 sys_catalog.cc:458] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:55:09.965229 23517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0: No bootstrap required, opened a new log
I20260809 01:55:09.965828 23399 sys_catalog.cc:564] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [sys.catalog]: configured and running, proceeding with master startup.
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
I20260809 01:55:09.969376 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.949197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.3.220.62:35817)
0809 01:55:09.949907 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.949911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.949923 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:55:09.949956 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.949959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.949996 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.950074 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.951202 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.951919 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.953009 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.953351 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.953437 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.956782 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.956787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.956793 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.956828 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.966360 (+ 9532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.966985 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.966989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.966992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.967084 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.967419 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.967423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.967426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.967702 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.967885 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.969031 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.969221 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":88,"threads_started":1}
W20260809 01:55:09.973119 23649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260809 01:55:09.973214 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.964231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:38395 (local address 127.0.0.1:58034)
0809 01:55:09.964357 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:09.964373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:09.964452 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:09.964738 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:09.964741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:09.964899 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:09.964992 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.964999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.966004 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.966007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.966799 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:09.966811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.967370 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.967373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:09.967444 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.967926 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:09.967940 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:09.969458 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:09.971659 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.971662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.971664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.971902 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.972196 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:09.972199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:09.972200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:09.972293 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:09.972754 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:09.972757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:09.972836 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 01:55:09.972916 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 01:55:09.973047 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 01:55:09.973495 23649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:55:09.973877 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.964442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.3.220.60:38395)
0809 01:55:09.964537 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.964542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.964554 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:55:09.964586 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.964589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.964629 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.965096 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.965175 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.965889 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.966936 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.967253 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.969615 (+ 2362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.969723 (+ 108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.969727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.969732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.969751 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.971555 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.972009 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.972012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.972013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.972091 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.972400 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.972405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.972410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.972643 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.973460 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.973607 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.973720 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 01:55:09.971611 23291 master_runner.cc:186] Error getting master registration for 127.3.220.61:37041: OK, instance_id { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" instance_seqno: 1786240509685398 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 01:55:09.975575 23291 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 09 01:55:09 dist-test-slave-gkw7 krb5kdc[23271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:55:09.969592 23517 raft_consensus.cc:348] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:09.977797 23517 raft_consensus.cc:374] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:09.977823 23621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92: Bootstrap starting.
I20260809 01:55:09.977878 23517 raft_consensus.cc:729] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28542ebfa7a4fd5a6b8ff060f207dc0, State: Initialized, Role: FOLLOWER
I20260809 01:55:09.978422 23517 consensus_queue.cc:260] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:09.979827 23517 sys_catalog.cc:564] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:09.980589 23621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:09.981585 23621 log.cc:824] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:09.982524 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.971790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.3.220.61:37041)
0809 01:55:09.971908 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.971913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.971926 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:55:09.971965 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.971971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.972018 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.972086 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.972974 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.973709 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.974814 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.975138 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.975246 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:09.978334 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:09.978338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:09.978341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:09.978370 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:09.980114 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.980630 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.980633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.980635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.980729 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:09.981041 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:09.981045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:09.981046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:09.981297 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:09.981488 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:09.982244 (+ 756us) server_negotiation.cc:299] Negotiation successful
0809 01:55:09.982383 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:55:09.985782 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:55:09.985800 23651 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:09.985932 23651 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:55:09.988416 23621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92: No bootstrap required, opened a new log
W20260809 01:55:09.990007 23670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:55:09.990094 23670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:55:09.992830 23621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:09.993096 23621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:09.993198 23621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efb47143a4448fc97291cceac7fcd92, State: Initialized, Role: FOLLOWER
I20260809 01:55:09.989312 23410 master_runner.cc:186] Error getting master registration for 127.3.220.60:38395: OK, instance_id { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" instance_seqno: 1786240509924055 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 01:55:09.993557 23410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:55:09.993753 23621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:09.995230 23621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:09.998028 23678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:09.998201 23678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:55:10.003859 23526 master_runner.cc:263] No leader master found from master 4efb47143a4448fc97291cceac7fcd92
I20260809 01:55:10.003937 23526 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:55:10.005417 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.993884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.3.220.61:37041)
0809 01:55:09.994013 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.994020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:09.995742 (+ 1722us) server_negotiation.cc:407] Connection header received
0809 01:55:09.995942 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:09.995946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:09.996001 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:09.996061 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:09.997207 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.998062 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.999209 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:09.999533 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:09.999606 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:10.001650 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:10.001657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:10.001660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:10.001687 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:10.003623 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:10.004112 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:10.004116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:10.004118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:10.004209 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:10.004502 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:10.004505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:10.004507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:10.004700 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:10.004857 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:10.005124 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 01:55:10.005257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:55:10.010242 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.994413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.3.220.62:35817)
0809 01:55:09.994549 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.994553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:10.001806 (+ 7253us) server_negotiation.cc:407] Connection header received
0809 01:55:10.002018 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:10.002024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:10.002078 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:10.002132 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:10.003060 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:10.003841 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:10.005017 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:10.005342 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:10.005447 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:10.007147 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:10.007150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:10.007153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:10.007180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:10.008495 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:10.008917 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:10.008921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:10.008923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:10.009008 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:10.009312 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:10.009316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:10.009317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:10.009504 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:10.009702 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:10.009917 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 01:55:10.010046 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 01:55:10.017071 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:09.995103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.3.220.60:38395)
0809 01:55:09.995210 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:09.995215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:10.005232 (+ 10017us) server_negotiation.cc:407] Connection header received
0809 01:55:10.005403 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:10.005406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:10.005453 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:10.005509 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:10.010251 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:10.010957 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:10.011898 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:10.012119 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:10.012236 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:10.013626 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:10.013630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:10.013633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:10.013672 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:10.015151 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:10.015640 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:10.015643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:10.015645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:10.015737 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:10.016049 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:10.016053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:10.016055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:10.016281 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:10.016413 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:10.016778 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 01:55:10.016903 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 01:55:10.017254 23698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:55:10.017331 23698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:55:10.043267 23651 raft_consensus.cc:482] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:10.043476 23651 raft_consensus.cc:504] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:10.044570 23651 leader_election.cc:288] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d479a314af154f7895d715ca614156ae (127.3.220.62:35817), 4efb47143a4448fc97291cceac7fcd92 (127.3.220.60:38395)
I20260809 01:55:10.045980 23355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d479a314af154f7895d715ca614156ae" is_pre_election: true
I20260809 01:55:10.046330 23355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28542ebfa7a4fd5a6b8ff060f207dc0 in term 0.
I20260809 01:55:10.046785 23437 leader_election.cc:302] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c28542ebfa7a4fd5a6b8ff060f207dc0, d479a314af154f7895d715ca614156ae; no voters:
I20260809 01:55:10.045382 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4efb47143a4448fc97291cceac7fcd92" is_pre_election: true
I20260809 01:55:10.047513 23583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28542ebfa7a4fd5a6b8ff060f207dc0 in term 0.
I20260809 01:55:10.048583 23651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:55:10.048696 23651 raft_consensus.cc:482] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:55:10.048753 23651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:10.050179 23651 raft_consensus.cc:504] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:10.050536 23651 leader_election.cc:288] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [CANDIDATE]: Term 1 election: Requested vote from peers d479a314af154f7895d715ca614156ae (127.3.220.62:35817), 4efb47143a4448fc97291cceac7fcd92 (127.3.220.60:38395)
I20260809 01:55:10.051126 23355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d479a314af154f7895d715ca614156ae"
I20260809 01:55:10.051260 23355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:10.052626 23355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28542ebfa7a4fd5a6b8ff060f207dc0 in term 1.
I20260809 01:55:10.052984 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4efb47143a4448fc97291cceac7fcd92"
I20260809 01:55:10.053115 23583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:10.053640 23437 leader_election.cc:302] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c28542ebfa7a4fd5a6b8ff060f207dc0, d479a314af154f7895d715ca614156ae; no voters:
I20260809 01:55:10.054685 23583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28542ebfa7a4fd5a6b8ff060f207dc0 in term 1.
I20260809 01:55:10.055419 23651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:10.055696 23651 raft_consensus.cc:686] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [term 1 LEADER]: Becoming Leader. State: Replica: c28542ebfa7a4fd5a6b8ff060f207dc0, State: Running, Role: LEADER
I20260809 01:55:10.056061 23651 consensus_queue.cc:237] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } }
I20260809 01:55:10.058213 23651 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28542ebfa7a4fd5a6b8ff060f207dc0. Latest consensus state: current_term: 1 leader_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:10.058337 23651 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:10.059684 23703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:10.064273 23703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:10.070948 23583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [term 1 FOLLOWER]: Refusing update from remote peer c28542ebfa7a4fd5a6b8ff060f207dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:10.074062 23701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:10.078435 23678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28542ebfa7a4fd5a6b8ff060f207dc0. Latest consensus state: current_term: 1 leader_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:10.078568 23678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:55:10.079612 23355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [term 1 FOLLOWER]: Refusing update from remote peer c28542ebfa7a4fd5a6b8ff060f207dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:10.080219 23701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:10.084419 23633 sys_catalog.cc:455] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28542ebfa7a4fd5a6b8ff060f207dc0. Latest consensus state: current_term: 1 leader_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:10.084553 23633 sys_catalog.cc:458] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:55:10.091061 23701 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:10.091204 23701 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:10.092280 23633 sys_catalog.cc:455] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:10.092403 23633 sys_catalog.cc:458] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:55:10.092785 23704 mvcc.cc:205] Tried to move back new op lower bound from 7316441129241698304 to 7316441129193451520. Current Snapshot: MvccSnapshot[applied={T|T < 7316441129241698304}]
I20260809 01:55:10.096315 23707 mvcc.cc:205] Tried to move back new op lower bound from 7316441129241698304 to 7316441129193451520. Current Snapshot: MvccSnapshot[applied={T|T < 7316441129241698304}]
I20260809 01:55:10.100644 23651 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:10.100790 23651 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:10.101907 23678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d479a314af154f7895d715ca614156ae" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35817 } } peers { permanent_uuid: "c28542ebfa7a4fd5a6b8ff060f207dc0" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37041 } } peers { permanent_uuid: "4efb47143a4448fc97291cceac7fcd92" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 38395 } } }
I20260809 01:55:10.102026 23678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:55:10.103111 23705 mvcc.cc:205] Tried to move back new op lower bound from 7316441129241698304 to 7316441129193451520. Current Snapshot: MvccSnapshot[applied={T|T < 7316441129241698304}]
I20260809 01:55:10.105033 23703 catalog_manager.cc:1347] Generated new cluster ID: 906889008d6545c695578d7f05d37b7b
I20260809 01:55:10.105100 23703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:55:10.138841 23703 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:55:10.140189 23703 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:55:10.160835 23703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c28542ebfa7a4fd5a6b8ff060f207dc0: Generated new TSK 0
I20260809 01:55:10.161757 23703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:55:10.975028 23649 catalog_manager.cc:1259] Loaded cluster ID: 906889008d6545c695578d7f05d37b7b
I20260809 01:55:10.975157 23649 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae: loading cluster ID for follower catalog manager: success
I20260809 01:55:10.976958 23649 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae: acquiring CA information for follower catalog manager: success
I20260809 01:55:10.978250 23649 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d479a314af154f7895d715ca614156ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:55:11.018759 23698 catalog_manager.cc:1259] Loaded cluster ID: 906889008d6545c695578d7f05d37b7b
I20260809 01:55:11.018901 23698 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92: loading cluster ID for follower catalog manager: success
I20260809 01:55:11.020663 23698 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92: acquiring CA information for follower catalog manager: success
I20260809 01:55:11.021934 23698 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4efb47143a4448fc97291cceac7fcd92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:55:11.234973 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 23291
W20260809 01:55:11.249794 23437 connection.cc:535] client connection to 127.3.220.62:35817 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 01:55:11.250065 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 23410
I20260809 01:55:11.265960 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 23526
W20260809 01:55:11.265959 23549 connection.cc:535] server connection from 127.0.0.1:58022 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
2026-08-09T01:55:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3350 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:55:11 dist-test-slave-gkw7 krb5kdc[23723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:55:11 dist-test-slave-gkw7 krb5kdc[23723](info): set up 2 sockets
Aug 09 01:55:11 dist-test-slave-gkw7 krb5kdc[23723](info): commencing operation
krb5kdc: starting...
W20260809 01:55:12.502977 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:55:12 dist-test-slave-gkw7 krb5kdc[23723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:55:12.634723 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:40937
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:40937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:12.755525 23742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:12.755867 23742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:12.755949 23742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:12.760066 23742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:12.760177 23742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:12.760222 23742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:12.760288 23742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:12.760330 23742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:12.766130 23742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:40937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:12.767304 23742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:12.768391 23742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:12.777311 23749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:12.777299 23748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:12.777482 23742 server_base.cc:1034] running on GCE node
W20260809 01:55:12.777319 23753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:12.778360 23742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:12.779556 23742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:12.780779 23742 hybrid_clock.cc:648] HybridClock initialized: now 1786240512780743 us; error 50 us; skew 500 ppm
Aug 09 01:55:12 dist-test-slave-gkw7 krb5kdc[23723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:12.788262 23742 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:12.789318 23742 webserver.cc:460] Webserver started at http://127.3.220.62:36703/ using document root <none> and password file <none>
I20260809 01:55:12.790040 23742 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:12.790159 23742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:12.790431 23742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:12.792838 23742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "4b2d4f7ebc2b460190781189ee07af0c"
format_stamp: "Formatted at 2026-08-09 01:55:12 on dist-test-slave-gkw7"
server_key: "9415305c63ed1801e098f6a3d0c51838"
server_key_iv: "b9d533bd4284370e8cbb8269228215d7"
server_key_version: "encryptionkey@0"
I20260809 01:55:12.793517 23742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "4b2d4f7ebc2b460190781189ee07af0c"
format_stamp: "Formatted at 2026-08-09 01:55:12 on dist-test-slave-gkw7"
server_key: "9415305c63ed1801e098f6a3d0c51838"
server_key_iv: "b9d533bd4284370e8cbb8269228215d7"
server_key_version: "encryptionkey@0"
I20260809 01:55:12.797900 23742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 01:55:12.800802 23765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:12.802229 23742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:55:12.802397 23742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "4b2d4f7ebc2b460190781189ee07af0c"
format_stamp: "Formatted at 2026-08-09 01:55:12 on dist-test-slave-gkw7"
server_key: "9415305c63ed1801e098f6a3d0c51838"
server_key_iv: "b9d533bd4284370e8cbb8269228215d7"
server_key_version: "encryptionkey@0"
I20260809 01:55:12.802521 23742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:12.815850 23742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:12.819278 23742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:12.819530 23742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:12.826722 23742 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:40937
I20260809 01:55:12.826795 23853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:40937 every 8 connection(s)
I20260809 01:55:12.827750 23742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:12.830905 23854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:12.832962 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 23742
I20260809 01:55:12.833105 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:55:12.833418 3952 external_mini_cluster.cc:1428] Setting key be3f1a7649c7322bcab2dc89faef3212
I20260809 01:55:12.838157 23854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c: Bootstrap starting.
I20260809 01:55:12.840692 23854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:12.841573 23854 log.cc:824] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:55:12 dist-test-slave-gkw7 krb5kdc[23723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:12.844044 23854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c: No bootstrap required, opened a new log
I20260809 01:55:12.849138 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:12.834972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.3.220.62:40937)
0809 01:55:12.835487 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:12.835509 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:12.835545 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:55:12.836226 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:12.836243 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:12.836585 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:12.836988 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:12.837929 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:12.838969 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:12.839873 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:12.840342 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:12.840519 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:12.843488 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:12.843494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:12.843511 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:12.843545 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:12.845684 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:12.846183 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:12.846189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:12.846196 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:12.846368 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:12.846711 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:12.846716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:12.846719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:12.847312 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:12.847507 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:12.847801 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 01:55:12.848120 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":169,"threads_started":1}
I20260809 01:55:12.849062 23854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2d4f7ebc2b460190781189ee07af0c" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40937 } }
I20260809 01:55:12.849350 23854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:12.849413 23854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2d4f7ebc2b460190781189ee07af0c, State: Initialized, Role: FOLLOWER
I20260809 01:55:12.849923 23854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2d4f7ebc2b460190781189ee07af0c" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40937 } }
I20260809 01:55:12.850080 23854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:55:12.850172 23854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:55:12.850274 23854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:12.851406 23854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2d4f7ebc2b460190781189ee07af0c" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40937 } }
I20260809 01:55:12.851823 23854 leader_election.cc:302] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b2d4f7ebc2b460190781189ee07af0c; no voters:
I20260809 01:55:12.852170 23854 leader_election.cc:288] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:55:12.852309 23860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:12.852514 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:55:12.852615 23860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [term 1 LEADER]: Becoming Leader. State: Replica: 4b2d4f7ebc2b460190781189ee07af0c, State: Running, Role: LEADER
I20260809 01:55:12.852921 23860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2d4f7ebc2b460190781189ee07af0c" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40937 } }
I20260809 01:55:12.853500 23854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:12.854310 23860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2d4f7ebc2b460190781189ee07af0c. Latest consensus state: current_term: 1 leader_uuid: "4b2d4f7ebc2b460190781189ee07af0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2d4f7ebc2b460190781189ee07af0c" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40937 } } }
I20260809 01:55:12.854444 23860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:12.855069 23862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b2d4f7ebc2b460190781189ee07af0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2d4f7ebc2b460190781189ee07af0c" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40937 } } }
I20260809 01:55:12.855188 23862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:12.855536 23869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:12.858871 23869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:12.864884 23869 catalog_manager.cc:1347] Generated new cluster ID: 6481869e9cab4af59708a82960dfa6d0
I20260809 01:55:12.864971 23869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:55:12.873315 23869 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:55:12.874267 23869 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:55:12.880088 23869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b2d4f7ebc2b460190781189ee07af0c: Generated new TSK 0
I20260809 01:55:12.880756 23869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:55:12.909482 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 23742
2026-08-09T01:55:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1647 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:55:12 dist-test-slave-gkw7 krb5kdc[23893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:55:12 dist-test-slave-gkw7 krb5kdc[23893](info): set up 2 sockets
Aug 09 01:55:12 dist-test-slave-gkw7 krb5kdc[23893](info): commencing operation
krb5kdc: starting...
W20260809 01:55:14.149995 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:55:14 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:55:14.275772 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39463
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:39463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:14.389927 23914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:14.390211 23914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:14.390254 23914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:14.394271 23914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:14.394328 23914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:14.394349 23914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:14.394366 23914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:14.394381 23914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:14.399811 23914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:39463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:14.400919 23914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:14.402017 23914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:14.410986 23923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:14.410933 23927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:14.410933 23925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:14.411273 23914 server_base.cc:1034] running on GCE node
I20260809 01:55:14.411830 23914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:14.412959 23914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:14.414114 23914 hybrid_clock.cc:648] HybridClock initialized: now 1786240514414085 us; error 41 us; skew 500 ppm
Aug 09 01:55:14 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:14.417834 23914 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:14.419054 23914 webserver.cc:460] Webserver started at http://127.3.220.62:41853/ using document root <none> and password file <none>
I20260809 01:55:14.419792 23914 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:14.419888 23914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:14.420209 23914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:14.422933 23914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "00a661d5bfe84ee3821337014243f5ec"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "9ed7b90d4cfa7c359816ae4154fc681f"
server_key_iv: "d1323df3fcfc565965166886fecfb468"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.423643 23914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "00a661d5bfe84ee3821337014243f5ec"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "9ed7b90d4cfa7c359816ae4154fc681f"
server_key_iv: "d1323df3fcfc565965166886fecfb468"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.428206 23914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:55:14.430785 23938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:14.432180 23914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 01:55:14.432343 23914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "00a661d5bfe84ee3821337014243f5ec"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "9ed7b90d4cfa7c359816ae4154fc681f"
server_key_iv: "d1323df3fcfc565965166886fecfb468"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.432454 23914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:14.442762 23914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:14.445711 23914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:14.445904 23914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:14.452908 23914 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:39463
I20260809 01:55:14.453204 24019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:39463 every 8 connection(s)
I20260809 01:55:14.453934 23914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:14.456688 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:14.462208 24022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Bootstrap starting.
I20260809 01:55:14.464125 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 23914
I20260809 01:55:14.464275 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:55:14.464594 3952 external_mini_cluster.cc:1428] Setting key b4fd932766d0561fb23c846b7ed64235
I20260809 01:55:14.464865 24022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:14.465808 24022 log.cc:824] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:14.467751 24022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: No bootstrap required, opened a new log
I20260809 01:55:14.471748 24022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:14.472038 24022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:14.472102 24022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a661d5bfe84ee3821337014243f5ec, State: Initialized, Role: FOLLOWER
I20260809 01:55:14.472630 24022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:14.472813 24022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:55:14.472898 24022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:55:14.473013 24022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 0 FOLLOWER]: Advancing to term 1
Aug 09 01:55:14 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:14.473950 24022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:14.474275 24022 leader_election.cc:302] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a661d5bfe84ee3821337014243f5ec; no voters:
I20260809 01:55:14.474596 24022 leader_election.cc:288] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:55:14.474699 24029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:14.474953 24029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 1 LEADER]: Becoming Leader. State: Replica: 00a661d5bfe84ee3821337014243f5ec, State: Running, Role: LEADER
I20260809 01:55:14.475232 24029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:14.475770 24022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:14.476819 24032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a661d5bfe84ee3821337014243f5ec. Latest consensus state: current_term: 1 leader_uuid: "00a661d5bfe84ee3821337014243f5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } } }
I20260809 01:55:14.476982 24032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:14.477330 24031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a661d5bfe84ee3821337014243f5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } } }
I20260809 01:55:14.477483 24031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:14.477398 24043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:14.480712 24043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:14.481710 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:14.465775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.3.220.62:39463)
0809 01:55:14.466228 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:14.466243 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:14.466277 (+ 34us) server_negotiation.cc:407] Connection header received
0809 01:55:14.466896 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:14.466912 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:14.467250 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:14.467526 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:14.468445 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:14.469437 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:14.470732 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:14.471183 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:14.471321 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:14.474094 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:14.474101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:14.474117 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:14.474154 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:14.478179 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:14.478615 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:14.478622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:14.478627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:14.478763 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:14.479072 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:14.479075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:14.479076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:14.479586 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:14.479788 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:14.480093 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 01:55:14.480400 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":173,"threads_started":1}
I20260809 01:55:14.486465 24043 catalog_manager.cc:1347] Generated new cluster ID: b550f8ea33734f6c8a53dfb2ff65a3c8
I20260809 01:55:14.486532 24043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:55:14.516505 24043 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:55:14.517831 24043 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:55:14.533315 24043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Generated new TSK 0
I20260809 01:55:14.534219 24043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:55:14.565641 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39463
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:14.678933 24059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:14.679198 24059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:14.679245 24059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:14.683609 24059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:14.683681 24059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:14.683739 24059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:55:14.688871 24059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:14.689936 24059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:14.690941 24059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:14.700943 24069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:14.700943 24067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:14.700948 24066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:14.701260 24059 server_base.cc:1034] running on GCE node
I20260809 01:55:14.701774 24059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:14.702425 24059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:14.703617 24059 hybrid_clock.cc:648] HybridClock initialized: now 1786240514703601 us; error 32 us; skew 500 ppm
Aug 09 01:55:14 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:14.706552 24059 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:55:14.707471 24059 webserver.cc:460] Webserver started at http://127.3.220.1:33847/ using document root <none> and password file <none>
I20260809 01:55:14.708024 24059 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:14.708112 24059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:14.708398 24059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:14.710546 24059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "92ef5cf560094a71ba808c7992215b22"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "6f178e85f4a168ffe903c81f58cb405e"
server_key_iv: "91f48bfcdf649da03a6320bc49446b05"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.711149 24059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "92ef5cf560094a71ba808c7992215b22"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "6f178e85f4a168ffe903c81f58cb405e"
server_key_iv: "91f48bfcdf649da03a6320bc49446b05"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.714982 24059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:55:14.717573 24077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:14.718811 24059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:55:14.718948 24059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "92ef5cf560094a71ba808c7992215b22"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "6f178e85f4a168ffe903c81f58cb405e"
server_key_iv: "91f48bfcdf649da03a6320bc49446b05"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.719058 24059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:14.751013 24059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:14.754077 24059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:14.754294 24059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:14.754909 24059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:14.755738 24059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:14.755807 24059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:14.755883 24059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:14.755924 24059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:14.765223 24059 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:46701
I20260809 01:55:14.765246 24253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:46701 every 8 connection(s)
I20260809 01:55:14.766142 24059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:55:14.772468 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 24059
I20260809 01:55:14.772617 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:55:14.772894 3952 external_mini_cluster.cc:1428] Setting key 453da4afde8b42d5c329e23572e16a74
Aug 09 01:55:14 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:14.780685 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:14.767931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:43453 (local address 127.3.220.62:39463)
0809 01:55:14.768141 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:14.768147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:14.769157 (+ 1010us) server_negotiation.cc:407] Connection header received
0809 01:55:14.770064 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:14.770067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:14.770113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:14.770188 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:14.771340 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:14.771867 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:14.773097 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:14.773398 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:14.773500 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:14.776584 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:14.776589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:14.776593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:14.776625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:14.778484 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:14.779000 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:14.779006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:14.779011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:14.779105 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:14.779441 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:14.779447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:14.779452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:14.779713 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:14.779827 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:14.780319 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 01:55:14.780466 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 01:55:14.781329 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:14.768365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:43453)
0809 01:55:14.769004 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:14.769046 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:14.769868 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:14.770368 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:14.770376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:14.770764 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:14.771218 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:14.771230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:14.771967 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:14.771970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:14.772984 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:14.772994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:14.773694 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:14.773697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:14.773849 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:14.774447 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:14.774463 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:14.776288 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:14.778606 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:14.778613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:14.778629 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:14.778900 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:14.779205 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:14.779208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:14.779209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:14.779343 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:14.779834 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:14.779841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:14.780066 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 01:55:14.780236 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 01:55:14.780488 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":113,"threads_started":1}
I20260809 01:55:14.782590 24254 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39463
I20260809 01:55:14.782814 24254 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:14.783280 24254 heartbeater.cc:511] Master 127.3.220.62:39463 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:55:14.784739 23964 ts_manager.cc:194] Registered new tserver with Master: 92ef5cf560094a71ba808c7992215b22 (127.3.220.1:46701)
I20260809 01:55:14.785960 23964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:43453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:55:14.806437 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39463
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:14.920298 24262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:14.920557 24262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:14.920596 24262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:14.924613 24262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:14.924671 24262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:14.924727 24262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:55:14.929791 24262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:14.930797 24262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:14.931813 24262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:14.941807 24272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:14.941814 24268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:14.941824 24267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:14.942008 24262 server_base.cc:1034] running on GCE node
I20260809 01:55:14.942541 24262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:14.943219 24262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:14.944397 24262 hybrid_clock.cc:648] HybridClock initialized: now 1786240514944368 us; error 45 us; skew 500 ppm
Aug 09 01:55:14 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:14.947151 24262 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:55:14.947964 24262 webserver.cc:460] Webserver started at http://127.3.220.2:41063/ using document root <none> and password file <none>
I20260809 01:55:14.948544 24262 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:14.948630 24262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:14.948858 24262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:14.950929 24262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "3634b4bcc1b24e3d868556cccf946d7b"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "485f25fd351d766434863c3eb09e5b79"
server_key_iv: "6e08593097a67fa8e6b6d772ac74ceb6"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.951493 24262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "3634b4bcc1b24e3d868556cccf946d7b"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "485f25fd351d766434863c3eb09e5b79"
server_key_iv: "6e08593097a67fa8e6b6d772ac74ceb6"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.955214 24262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260809 01:55:14.957665 24281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:14.958848 24262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:55:14.958977 24262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "3634b4bcc1b24e3d868556cccf946d7b"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "485f25fd351d766434863c3eb09e5b79"
server_key_iv: "6e08593097a67fa8e6b6d772ac74ceb6"
server_key_version: "encryptionkey@0"
I20260809 01:55:14.959101 24262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:14.988324 24262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:14.991694 24262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:14.991904 24262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:14.992527 24262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:14.993359 24262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:14.993427 24262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:14.993494 24262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:14.993564 24262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:15.002345 24262 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:42525
I20260809 01:55:15.002382 24452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:42525 every 8 connection(s)
I20260809 01:55:15.003237 24262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:55:15.003878 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 24262
I20260809 01:55:15.004019 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:55:15.004287 3952 external_mini_cluster.cc:1428] Setting key 62750fd71f375c4e1eac16149ab47153
Aug 09 01:55:15 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240514, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:55:15.018496 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.005119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:58673 (local address 127.3.220.62:39463)
0809 01:55:15.005262 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.005267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.006659 (+ 1392us) server_negotiation.cc:407] Connection header received
0809 01:55:15.008011 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.008037 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.008106 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.008220 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:15.009361 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.010256 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.011480 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.011914 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.012053 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.014893 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:15.014896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:15.014898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:15.014922 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:15.016427 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.016948 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.016951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.016952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.017026 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.017328 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.017348 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.017350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.017519 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:15.017681 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.018214 (+ 533us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.018319 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:55:15.019428 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.005648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:58673)
0809 01:55:15.006476 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.006534 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.007782 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.008411 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.008417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.008758 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:15.009217 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.009230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.010382 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.010385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.011353 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.011363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.012065 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.012068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.012210 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.012850 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:15.012866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:15.014718 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:15.016570 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:15.016579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:15.016596 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:15.016861 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:15.017116 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:15.017118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:15.017119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:15.017248 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:15.017725 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:15.017731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:15.017995 (+ 264us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.018164 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.018530 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":150,"threads_started":1}
I20260809 01:55:15.020717 24453 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39463
I20260809 01:55:15.020952 24453 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:15.021483 24453 heartbeater.cc:511] Master 127.3.220.62:39463 requested a full tablet report, sending...
I20260809 01:55:15.022631 23964 ts_manager.cc:194] Registered new tserver with Master: 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525)
I20260809 01:55:15.023224 23964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:58673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:55:15.036701 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39463
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:15.148998 24463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:15.149273 24463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:15.149346 24463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:15.153260 24463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:15.153337 24463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:15.153427 24463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:55:15.158507 24463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:15.159576 24463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:15.160635 24463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:15.170569 24471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:15.170739 24463 server_base.cc:1034] running on GCE node
W20260809 01:55:15.170802 24473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:15.170759 24475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:15.171271 24463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:15.171888 24463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:15.173082 24463 hybrid_clock.cc:648] HybridClock initialized: now 1786240515173064 us; error 33 us; skew 500 ppm
Aug 09 01:55:15 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:15.176004 24463 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:55:15.176972 24463 webserver.cc:460] Webserver started at http://127.3.220.3:42167/ using document root <none> and password file <none>
I20260809 01:55:15.177560 24463 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:15.177649 24463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:15.177932 24463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:15.179977 24463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "1c19d7c8623f4b9da3ea6d735e1090e9"
format_stamp: "Formatted at 2026-08-09 01:55:15 on dist-test-slave-gkw7"
server_key: "85666a6eac25ef6d99893a24d2e914ef"
server_key_iv: "263be5e6a963c5fc3047a4dca2931b9b"
server_key_version: "encryptionkey@0"
I20260809 01:55:15.180562 24463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "1c19d7c8623f4b9da3ea6d735e1090e9"
format_stamp: "Formatted at 2026-08-09 01:55:15 on dist-test-slave-gkw7"
server_key: "85666a6eac25ef6d99893a24d2e914ef"
server_key_iv: "263be5e6a963c5fc3047a4dca2931b9b"
server_key_version: "encryptionkey@0"
I20260809 01:55:15.184381 24463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 01:55:15.186949 24483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:15.188337 24463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 01:55:15.188484 24463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "1c19d7c8623f4b9da3ea6d735e1090e9"
format_stamp: "Formatted at 2026-08-09 01:55:15 on dist-test-slave-gkw7"
server_key: "85666a6eac25ef6d99893a24d2e914ef"
server_key_iv: "263be5e6a963c5fc3047a4dca2931b9b"
server_key_version: "encryptionkey@0"
I20260809 01:55:15.188594 24463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:15.205832 24463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:15.208884 24463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:15.209098 24463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:15.209729 24463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:15.210566 24463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:15.210614 24463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:15.210685 24463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:15.210722 24463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:15.219070 24463 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:39527
I20260809 01:55:15.219137 24648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:39527 every 8 connection(s)
I20260809 01:55:15.219974 24463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:55:15.224072 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 24463
I20260809 01:55:15.224228 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:55:15.224509 3952 external_mini_cluster.cc:1428] Setting key af4c4044860fc547b3a3100ef8c33ec5
Aug 09 01:55:15 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:15.233788 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.221758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:45255 (local address 127.3.220.62:39463)
0809 01:55:15.221960 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.221966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.223068 (+ 1102us) server_negotiation.cc:407] Connection header received
0809 01:55:15.223966 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.223969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.224028 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.224130 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:15.225632 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.226228 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.227526 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.227739 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.227832 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.230226 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:15.230229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:15.230231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:15.230255 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:15.231985 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.232450 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.232452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.232454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.232527 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.232806 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.232808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.232810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.233001 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:15.233081 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.233516 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.233664 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260809 01:55:15.234649 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.222221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:45255)
0809 01:55:15.222936 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.222978 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.223723 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.224328 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.224339 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.224752 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:15.225451 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.225475 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.226355 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.226359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.227419 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.227429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.227872 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.227875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.228011 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.228544 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:15.228560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:15.230075 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:15.232088 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:15.232093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:15.232105 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:15.232358 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:15.232598 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:15.232601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:15.232602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:15.232727 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:15.233084 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:15.233089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:15.233308 (+ 219us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.233456 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.233760 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":105,"threads_started":1}
I20260809 01:55:15.235697 24651 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39463
I20260809 01:55:15.235941 24651 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:15.236414 24651 heartbeater.cc:511] Master 127.3.220.62:39463 requested a full tablet report, sending...
I20260809 01:55:15.237609 23964 ts_manager.cc:194] Registered new tserver with Master: 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
I20260809 01:55:15.238266 23964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:45255
I20260809 01:55:15.238667 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:55:15.248087 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.240883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.3.220.62:39463)
0809 01:55:15.240997 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.241001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.241225 (+ 224us) server_negotiation.cc:407] Connection header received
0809 01:55:15.241374 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.241377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.241418 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.241501 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:15.242289 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.242783 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.243638 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.243866 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.243965 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.245216 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:15.245219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:15.245222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:15.245246 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:15.246466 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.246869 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.246874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.246876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.246957 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.247221 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.247226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.247228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.247440 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:15.247579 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.247814 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.247925 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 01:55:15.250775 23964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:55:15.253782 23964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:55:15.267634 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.261709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:42525 (local address 127.0.0.1:34580)
0809 01:55:15.262195 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.262241 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.262380 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.263720 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.263727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.263749 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:15.263909 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.263917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.264976 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.264981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.266245 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.266254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.267087 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.267090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.267194 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.267241 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.267370 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.267482 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":40,"thread_start_us":96,"threads_started":1}
I20260809 01:55:15.267661 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.261873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.3.220.2:42525)
0809 01:55:15.263276 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.263291 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.263322 (+ 31us) server_negotiation.cc:407] Connection header received
0809 01:55:15.263393 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.263397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.263575 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.263728 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:15.264064 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.264851 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.266366 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.266962 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.267071 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.267214 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.267412 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.267511 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":153,"threads_started":1}
I20260809 01:55:15.268554 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.262470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46701 (local address 127.0.0.1:58514)
0809 01:55:15.263380 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.263393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.263473 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.263987 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.263991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.264013 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:15.264095 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.264102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.265522 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.265525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.266841 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.266852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.268071 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.268075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.268203 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.268270 (+ 67us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.268366 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.268442 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":196,"thread_start_us":94,"threads_started":1}
I20260809 01:55:15.268648 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.262571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.3.220.3:39527)
0809 01:55:15.262990 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.263003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.263162 (+ 159us) server_negotiation.cc:407] Connection header received
0809 01:55:15.263316 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.263336 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.263578 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.263725 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:15.264155 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.265375 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.266926 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.267742 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.267845 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.267968 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.268418 (+ 450us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.268513 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":185,"threads_started":1}
I20260809 01:55:15.268987 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.262829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.3.220.1:46701)
0809 01:55:15.263152 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.263166 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.263566 (+ 400us) server_negotiation.cc:407] Connection header received
0809 01:55:15.263653 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.263660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.263845 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.264000 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:15.264188 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.265360 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.266967 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.267937 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.268061 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.268197 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.268748 (+ 551us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.268849 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":125,"threads_started":1}
I20260809 01:55:15.269019 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.261872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:39527 (local address 127.0.0.1:33488)
0809 01:55:15.263065 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.263079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.263183 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.263890 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.263895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.263919 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:15.264017 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.264024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.265527 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.265530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.266788 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.266799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.268035 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.268040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.268151 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.268266 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.268803 (+ 537us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.268883 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":71,"thread_start_us":98,"threads_started":1}
I20260809 01:55:15.271277 24356 tablet_service.cc:1467] Processing CreateTablet for tablet aeb4ac8ed52c4bfb8b2b05c628895abb (DEFAULT_TABLE table=test-table [id=148ae1fb0d1c49e3931f3f0093610ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:15.272260 24356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb4ac8ed52c4bfb8b2b05c628895abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:15.272013 24555 tablet_service.cc:1467] Processing CreateTablet for tablet aeb4ac8ed52c4bfb8b2b05c628895abb (DEFAULT_TABLE table=test-table [id=148ae1fb0d1c49e3931f3f0093610ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:15.272014 24144 tablet_service.cc:1467] Processing CreateTablet for tablet aeb4ac8ed52c4bfb8b2b05c628895abb (DEFAULT_TABLE table=test-table [id=148ae1fb0d1c49e3931f3f0093610ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:15.273046 24555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb4ac8ed52c4bfb8b2b05c628895abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:15.273037 24144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb4ac8ed52c4bfb8b2b05c628895abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:15.280615 24674 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: Bootstrap starting.
I20260809 01:55:15.281032 24675 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Bootstrap starting.
I20260809 01:55:15.283097 24679 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: Bootstrap starting.
I20260809 01:55:15.283663 24675 tablet_bootstrap.cc:654] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:15.284045 24674 tablet_bootstrap.cc:654] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:15.284580 24675 log.cc:824] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:15.284759 24674 log.cc:824] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:15.285578 24679 tablet_bootstrap.cc:654] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:15.286324 24679 log.cc:824] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:15.286715 24675 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: No bootstrap required, opened a new log
I20260809 01:55:15.286938 24675 ts_tablet_manager.cc:1397] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:55:15.288152 24679 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: No bootstrap required, opened a new log
I20260809 01:55:15.288369 24679 ts_tablet_manager.cc:1397] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:55:15.290678 24675 raft_consensus.cc:348] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.290915 24675 raft_consensus.cc:374] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:15.290988 24675 raft_consensus.cc:729] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ef5cf560094a71ba808c7992215b22, State: Initialized, Role: FOLLOWER
I20260809 01:55:15.291163 24674 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: No bootstrap required, opened a new log
I20260809 01:55:15.291370 24674 ts_tablet_manager.cc:1397] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260809 01:55:15.291502 24675 consensus_queue.cc:260] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.291749 24679 raft_consensus.cc:348] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.292146 24254 heartbeater.cc:503] Master 127.3.220.62:39463 was elected leader, sending a full tablet report...
I20260809 01:55:15.292160 24679 raft_consensus.cc:374] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:15.292222 24679 raft_consensus.cc:729] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3634b4bcc1b24e3d868556cccf946d7b, State: Initialized, Role: FOLLOWER
I20260809 01:55:15.292512 24675 ts_tablet_manager.cc:1428] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:55:15.292670 24679 consensus_queue.cc:260] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.293331 24453 heartbeater.cc:503] Master 127.3.220.62:39463 was elected leader, sending a full tablet report...
I20260809 01:55:15.293802 24679 ts_tablet_manager.cc:1428] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:55:15.295281 24674 raft_consensus.cc:348] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.295660 24674 raft_consensus.cc:374] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:15.295747 24674 raft_consensus.cc:729] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c19d7c8623f4b9da3ea6d735e1090e9, State: Initialized, Role: FOLLOWER
I20260809 01:55:15.296170 24674 consensus_queue.cc:260] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.296864 24651 heartbeater.cc:503] Master 127.3.220.62:39463 was elected leader, sending a full tablet report...
I20260809 01:55:15.297185 24674 ts_tablet_manager.cc:1428] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:55:15.335769 24686 raft_consensus.cc:482] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:15.335983 24686 raft_consensus.cc:504] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.337144 24686 leader_election.cc:288] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
I20260809 01:55:15.341557 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.337489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:57701 (local address 127.3.220.2:42525)
0809 01:55:15.337639 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.337645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.338003 (+ 358us) server_negotiation.cc:407] Connection header received
0809 01:55:15.338143 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.338145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.338206 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.338286 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:15.338581 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.339168 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.340505 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.340947 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.341028 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.341116 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.341323 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.341411 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 01:55:15.341566 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.337489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:59875 (local address 127.3.220.3:39527)
0809 01:55:15.337639 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.337645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.337934 (+ 289us) server_negotiation.cc:407] Connection header received
0809 01:55:15.338086 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.338090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.338150 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.338208 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:15.338811 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.339477 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.340576 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.340970 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.341089 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.341154 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.341347 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.341426 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 01:55:15.341598 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.337403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:42525 (local address 127.3.220.1:57701)
0809 01:55:15.337859 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.337874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.338024 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.338311 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.338314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.338356 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:15.338467 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.338472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.339286 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.339290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.340368 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.340382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.341086 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.341090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.341185 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.341214 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.341275 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.341420 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":22,"thread_start_us":164,"threads_started":1}
I20260809 01:55:15.342144 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.337403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:39527 (local address 127.3.220.1:59875)
0809 01:55:15.337843 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.337855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.337968 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.338242 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.338491 (+ 249us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.338507 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:15.338652 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.338659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.339591 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.339593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.340463 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.340476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.341105 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.341109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.341213 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.341230 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.341328 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.341963 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":566,"thread_start_us":130,"threads_started":1}
I20260809 01:55:15.342345 24383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb4ac8ed52c4bfb8b2b05c628895abb" candidate_uuid: "92ef5cf560094a71ba808c7992215b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3634b4bcc1b24e3d868556cccf946d7b" is_pre_election: true
I20260809 01:55:15.342684 24383 raft_consensus.cc:2454] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ef5cf560094a71ba808c7992215b22 in term 0.
I20260809 01:55:15.343101 24579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb4ac8ed52c4bfb8b2b05c628895abb" candidate_uuid: "92ef5cf560094a71ba808c7992215b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" is_pre_election: true
I20260809 01:55:15.343163 24080 leader_election.cc:302] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3634b4bcc1b24e3d868556cccf946d7b, 92ef5cf560094a71ba808c7992215b22; no voters:
I20260809 01:55:15.343389 24579 raft_consensus.cc:2454] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ef5cf560094a71ba808c7992215b22 in term 0.
I20260809 01:55:15.343446 24686 raft_consensus.cc:2781] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:55:15.343531 24686 raft_consensus.cc:482] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:55:15.343587 24686 raft_consensus.cc:3037] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:15.344732 24686 raft_consensus.cc:504] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.345103 24686 leader_election.cc:288] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 1 election: Requested vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
I20260809 01:55:15.345408 24579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb4ac8ed52c4bfb8b2b05c628895abb" candidate_uuid: "92ef5cf560094a71ba808c7992215b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9"
I20260809 01:55:15.345408 24383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb4ac8ed52c4bfb8b2b05c628895abb" candidate_uuid: "92ef5cf560094a71ba808c7992215b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3634b4bcc1b24e3d868556cccf946d7b"
I20260809 01:55:15.345551 24579 raft_consensus.cc:3037] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:15.345551 24383 raft_consensus.cc:3037] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:15.346881 24579 raft_consensus.cc:2454] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ef5cf560094a71ba808c7992215b22 in term 1.
I20260809 01:55:15.346881 24383 raft_consensus.cc:2454] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ef5cf560094a71ba808c7992215b22 in term 1.
I20260809 01:55:15.347159 24081 leader_election.cc:302] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c19d7c8623f4b9da3ea6d735e1090e9, 92ef5cf560094a71ba808c7992215b22; no voters:
I20260809 01:55:15.347330 24686 raft_consensus.cc:2781] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:15.347589 24686 raft_consensus.cc:686] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 1 LEADER]: Becoming Leader. State: Replica: 92ef5cf560094a71ba808c7992215b22, State: Running, Role: LEADER
I20260809 01:55:15.347908 24686 consensus_queue.cc:237] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.351841 23959 catalog_manager.cc:5495] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92ef5cf560094a71ba808c7992215b22 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "92ef5cf560094a71ba808c7992215b22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } health_report { overall_health: HEALTHY } } }
I20260809 01:55:15.369601 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.366357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.3.220.1:46701)
0809 01:55:15.366476 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.366479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.366493 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:55:15.366648 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.366654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.366729 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.366837 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:55:15.367056 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.367582 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.368342 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.368536 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.368608 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.368849 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:55:15.369251 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:55:15.369350 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.369401 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.369451 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 01:55:15.375308 24579 raft_consensus.cc:1261] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 FOLLOWER]: Refusing update from remote peer 92ef5cf560094a71ba808c7992215b22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:15.375308 24383 raft_consensus.cc:1261] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 1 FOLLOWER]: Refusing update from remote peer 92ef5cf560094a71ba808c7992215b22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:15.376080 24693 consensus_queue.cc:1035] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:15.376251 24686 consensus_queue.cc:1035] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:15.383247 24698 mvcc.cc:205] Tried to move back new op lower bound from 7316441150969864192 to 7316441150868115456. Current Snapshot: MvccSnapshot[applied={T|T < 7316441150969864192}]
I20260809 01:55:15.383425 24701 mvcc.cc:205] Tried to move back new op lower bound from 7316441150969864192 to 7316441150868115456. Current Snapshot: MvccSnapshot[applied={T|T < 7316441150969864192}]
I20260809 01:55:15.384979 24700 mvcc.cc:205] Tried to move back new op lower bound from 7316441150969864192 to 7316441150868115456. Current Snapshot: MvccSnapshot[applied={T|T < 7316441150969864192}]
I20260809 01:55:15.394254 23963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:15.394448 23963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:15.396947 23963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Sending DeleteTablet for 3 replicas of tablet aeb4ac8ed52c4bfb8b2b05c628895abb
I20260809 01:55:15.397776 24144 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb4ac8ed52c4bfb8b2b05c628895abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:15 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:58514
I20260809 01:55:15.397786 24356 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb4ac8ed52c4bfb8b2b05c628895abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:15 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:34580
I20260809 01:55:15.397976 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 24059
I20260809 01:55:15.398286 24555 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb4ac8ed52c4bfb8b2b05c628895abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:15 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:33488
I20260809 01:55:15.398731 24714 tablet_replica.cc:330] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: stopping tablet replica
I20260809 01:55:15.399075 24714 raft_consensus.cc:2229] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:15.399362 24714 raft_consensus.cc:2258] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:15.402130 24713 tablet_replica.cc:330] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: stopping tablet replica
I20260809 01:55:15.402438 24713 raft_consensus.cc:2229] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:15.402738 24713 raft_consensus.cc:2258] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:15.402931 24714 ts_tablet_manager.cc:1905] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:15.403980 24713 ts_tablet_manager.cc:1905] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:15.406173 24713 ts_tablet_manager.cc:1918] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:15.406291 24713 log.cc:1196] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/aeb4ac8ed52c4bfb8b2b05c628895abb
I20260809 01:55:15.406594 24714 ts_tablet_manager.cc:1918] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:15.406683 24713 ts_tablet_manager.cc:1939] T aeb4ac8ed52c4bfb8b2b05c628895abb P 3634b4bcc1b24e3d868556cccf946d7b: Deleting consensus metadata
I20260809 01:55:15.406702 24714 log.cc:1196] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/aeb4ac8ed52c4bfb8b2b05c628895abb
I20260809 01:55:15.407055 24714 ts_tablet_manager.cc:1939] T aeb4ac8ed52c4bfb8b2b05c628895abb P 1c19d7c8623f4b9da3ea6d735e1090e9: Deleting consensus metadata
I20260809 01:55:15.407927 23939 catalog_manager.cc:4841] TS 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525): tablet aeb4ac8ed52c4bfb8b2b05c628895abb (table test-table [id=148ae1fb0d1c49e3931f3f0093610ab2]) successfully deleted
I20260809 01:55:15.409659 23940 catalog_manager.cc:4841] TS 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527): tablet aeb4ac8ed52c4bfb8b2b05c628895abb (table test-table [id=148ae1fb0d1c49e3931f3f0093610ab2]) successfully deleted
W20260809 01:55:15.412509 23939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:46701 (error 108)
I20260809 01:55:15.412585 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 24262
I20260809 01:55:15.418603 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.418069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46701 (local address 127.0.0.1:58534)
0809 01:55:15.418338 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.418464 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:46701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260809 01:55:15.418831 23939 catalog_manager.cc:4584] TS 92ef5cf560094a71ba808c7992215b22 (127.3.220.1:46701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aeb4ac8ed52c4bfb8b2b05c628895abb: Network error: Client connection negotiation failed: client connection to 127.3.220.1:46701: connect: Connection refused (error 111)
I20260809 01:55:15.427711 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 24463
I20260809 01:55:15.441265 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 23914
I20260809 01:55:15.454018 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39463
--webserver_interface=127.3.220.62
--webserver_port=41853
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:39463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:15.567745 24720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:15.568018 24720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:15.568058 24720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:15.572093 24720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:15.572153 24720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:15.572211 24720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:15.572227 24720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:15.572242 24720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:15.577569 24720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:39463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=41853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:15.578672 24720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:15.579757 24720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:15.593789 24731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:15.593801 24729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:15.593801 24734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:15.594115 24720 server_base.cc:1034] running on GCE node
I20260809 01:55:15.594748 24720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:15.595868 24720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:15.597054 24720 hybrid_clock.cc:648] HybridClock initialized: now 1786240515597022 us; error 52 us; skew 500 ppm
Aug 09 01:55:15 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:15.599766 24720 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:15.600693 24720 webserver.cc:460] Webserver started at http://127.3.220.62:41853/ using document root <none> and password file <none>
I20260809 01:55:15.601275 24720 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:15.601385 24720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:15.605227 24720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:55:15.607411 24743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:15.608605 24720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 01:55:15.608747 24720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "00a661d5bfe84ee3821337014243f5ec"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "9ed7b90d4cfa7c359816ae4154fc681f"
server_key_iv: "d1323df3fcfc565965166886fecfb468"
server_key_version: "encryptionkey@0"
I20260809 01:55:15.609355 24720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:15.631136 24720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:15.634094 24720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:15.634311 24720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:15.641662 24720 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:39463
I20260809 01:55:15.641984 24815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:39463 every 8 connection(s)
I20260809 01:55:15.643082 24720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:15.647662 24817 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:55:15.649302 24817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Bootstrap starting.
I20260809 01:55:15.651444 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 24720
I20260809 01:55:15.651893 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:46701
--local_ip_for_outbound_sockets=127.3.220.1
--tserver_master_addrs=127.3.220.62:39463
--webserver_port=33847
--webserver_interface=127.3.220.1
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:15.671005 24817 log.cc:824] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:15.682515 24817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:55:15.682983 24817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Bootstrap complete.
I20260809 01:55:15.688650 24817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:15.689189 24817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a661d5bfe84ee3821337014243f5ec, State: Initialized, Role: FOLLOWER
I20260809 01:55:15.689778 24817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:15.689944 24817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:55:15.690019 24817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:55:15.690119 24817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:55:15.692580 24817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:15.692962 24817 leader_election.cc:302] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a661d5bfe84ee3821337014243f5ec; no voters:
I20260809 01:55:15.693342 24817 leader_election.cc:288] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 01:55:15.693468 24821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 2 FOLLOWER]: Leader election won for term 2
I20260809 01:55:15.693760 24821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 2 LEADER]: Becoming Leader. State: Replica: 00a661d5bfe84ee3821337014243f5ec, State: Running, Role: LEADER
I20260809 01:55:15.694132 24821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:15.694644 24817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:15.695696 24824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a661d5bfe84ee3821337014243f5ec. Latest consensus state: current_term: 2 leader_uuid: "00a661d5bfe84ee3821337014243f5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } } }
I20260809 01:55:15.695847 24824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:15.695696 24822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "00a661d5bfe84ee3821337014243f5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } } }
I20260809 01:55:15.696208 24822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:15.696229 24832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:15.701082 24832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:15.701937 24832 catalog_manager.cc:1259] Loaded cluster ID: b550f8ea33734f6c8a53dfb2ff65a3c8
I20260809 01:55:15.702014 24832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:55:15.703136 24832 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 01:55:15.705927 24832 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 01:55:15.710958 24832 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:55:15.711623 24832 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Loaded TSK: 0
I20260809 01:55:15.712596 24832 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 01:55:15.787243 24819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:15.787477 24819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:15.787530 24819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:15.791776 24819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:15.791839 24819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:15.791898 24819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:55:15.797228 24819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:46701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=33847
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:15.798249 24819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:15.799263 24819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:15.810174 24857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:15.810140 24849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:15.810418 24819 server_base.cc:1034] running on GCE node
W20260809 01:55:15.810140 24855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:15.811080 24819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:15.811772 24819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:15.812978 24819 hybrid_clock.cc:648] HybridClock initialized: now 1786240515812966 us; error 43 us; skew 500 ppm
Aug 09 01:55:15 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:15.816001 24819 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:55:15.816898 24819 webserver.cc:460] Webserver started at http://127.3.220.1:33847/ using document root <none> and password file <none>
I20260809 01:55:15.817467 24819 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:15.817590 24819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:15.821667 24819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 01:55:15.823916 24866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:15.825246 24819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:55:15.825387 24819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "92ef5cf560094a71ba808c7992215b22"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "6f178e85f4a168ffe903c81f58cb405e"
server_key_iv: "91f48bfcdf649da03a6320bc49446b05"
server_key_version: "encryptionkey@0"
I20260809 01:55:15.825994 24819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:15.847309 24819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:15.850354 24819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:15.850550 24819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:15.851161 24819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:15.852289 24877 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 01:55:15.856455 24819 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 01:55:15.856525 24819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 01:55:15.856617 24819 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 01:55:15.858315 24819 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 01:55:15.858381 24819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260809 01:55:15.858709 24877 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Bootstrap starting.
I20260809 01:55:15.868948 24819 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:46701
I20260809 01:55:15.869079 25033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:46701 every 8 connection(s)
I20260809 01:55:15.870029 24819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:55:15.870780 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 24819
I20260809 01:55:15.871202 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:42525
--local_ip_for_outbound_sockets=127.3.220.2
--tserver_master_addrs=127.3.220.62:39463
--webserver_port=41063
--webserver_interface=127.3.220.2
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:15.879678 24877 log.cc:824] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:55:15 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:15.887909 24877 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:55:15.888453 24877 tablet_bootstrap.cc:492] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Bootstrap complete.
I20260809 01:55:15.889946 24877 ts_tablet_manager.cc:1397] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.012s
I20260809 01:55:15.891358 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.873135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:56813 (local address 127.3.220.62:39463)
0809 01:55:15.873579 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.873600 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.874672 (+ 1072us) server_negotiation.cc:407] Connection header received
0809 01:55:15.876260 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.876270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.876590 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.876928 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:15.878099 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.879158 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.880532 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.881377 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.881639 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.884973 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:15.884980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:15.884997 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:15.885035 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:15.887266 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.888103 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.888112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.888118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.888281 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.888773 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.888779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.888784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.889273 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:15.889416 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.890201 (+ 785us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.890520 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":134,"threads_started":1}
I20260809 01:55:15.891363 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.873145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:56813)
0809 01:55:15.874477 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.874537 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.875857 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.876971 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.876979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.877411 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:15.877923 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.877937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.879331 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.879334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.880378 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.880389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.881585 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.881590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.881775 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.882529 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:15.882551 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:15.884729 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:15.887417 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:15.887424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:15.887440 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:15.887891 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:15.888399 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:15.888403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:15.888405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:15.888604 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:15.889402 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:15.889409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:15.889782 (+ 373us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.890006 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.890367 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"thread_start_us":154,"threads_started":1}
I20260809 01:55:15.892824 24877 raft_consensus.cc:348] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.893358 24877 raft_consensus.cc:729] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ef5cf560094a71ba808c7992215b22, State: Initialized, Role: FOLLOWER
I20260809 01:55:15.893939 24877 consensus_queue.cc:260] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:15.894209 25037 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39463
I20260809 01:55:15.894461 25037 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:15.894925 24877 ts_tablet_manager.cc:1428] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:55:15.895104 25037 heartbeater.cc:511] Master 127.3.220.62:39463 requested a full tablet report, sending...
I20260809 01:55:15.897809 24766 ts_manager.cc:194] Registered new tserver with Master: 92ef5cf560094a71ba808c7992215b22 (127.3.220.1:46701)
I20260809 01:55:15.901130 24766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:56813
I20260809 01:55:15.903903 25037 heartbeater.cc:503] Master 127.3.220.62:39463 was elected leader, sending a full tablet report...
Aug 09 01:55:15 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:55:15.911235 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.900632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46701 (local address 127.0.0.1:58550)
0809 01:55:15.901205 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:15.901252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:15.901398 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:15.902106 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:15.902112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:15.902481 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:15.902583 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.902590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.903538 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.903543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.904250 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:15.904277 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.905065 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.905068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:15.905172 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.905815 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:15.905832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:15.907392 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:15.909638 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:15.909646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:15.909650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:15.909859 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:15.910177 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:15.910179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:15.910180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:15.910251 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:15.910762 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:15.910767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:15.910860 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 01:55:15.910968 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 01:55:15.911108 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":124,"threads_started":1}
I20260809 01:55:15.911707 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:15.900712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.3.220.1:46701)
0809 01:55:15.901441 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:15.901456 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:15.901482 (+ 26us) server_negotiation.cc:407] Connection header received
0809 01:55:15.901607 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:15.901613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:15.901819 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:15.901975 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:15.902712 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.903427 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.904407 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:15.904918 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:15.905046 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:15.907538 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:15.907550 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:15.907563 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:15.907600 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:15.909477 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.909972 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.909978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.909980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.910082 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:15.910335 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:15.910339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:15.910340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:15.910651 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:15.910794 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:15.911077 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 01:55:15.911252 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":124,"threads_started":1}
I20260809 01:55:15.912642 24941 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb4ac8ed52c4bfb8b2b05c628895abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:15 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:58550
I20260809 01:55:15.913574 25048 tablet_replica.cc:330] stopping tablet replica
I20260809 01:55:15.913888 25048 raft_consensus.cc:2229] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:15.914135 25048 raft_consensus.cc:2258] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:15.915108 25048 ts_tablet_manager.cc:1905] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:15.918499 25048 ts_tablet_manager.cc:1918] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:15.918582 25048 log.cc:1196] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/aeb4ac8ed52c4bfb8b2b05c628895abb
I20260809 01:55:15.918862 25048 ts_tablet_manager.cc:1939] T aeb4ac8ed52c4bfb8b2b05c628895abb P 92ef5cf560094a71ba808c7992215b22: Deleting consensus metadata
I20260809 01:55:15.920526 24744 catalog_manager.cc:4841] TS 92ef5cf560094a71ba808c7992215b22 (127.3.220.1:46701): tablet aeb4ac8ed52c4bfb8b2b05c628895abb (table test-table [id=148ae1fb0d1c49e3931f3f0093610ab2]) successfully deleted
W20260809 01:55:16.000201 25040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:16.000502 25040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:16.000581 25040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:16.004675 25040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:16.004773 25040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:16.004868 25040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:55:16.010046 25040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:42525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=41063
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:16.011144 25040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:16.012202 25040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:16.022208 25056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:16.022181 25040 server_base.cc:1034] running on GCE node
W20260809 01:55:16.022192 25058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:16.022192 25055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:16.022800 25040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:16.023430 25040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:16.024605 25040 hybrid_clock.cc:648] HybridClock initialized: now 1786240516024583 us; error 30 us; skew 500 ppm
Aug 09 01:55:16 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:16.027410 25040 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:55:16.028338 25040 webserver.cc:460] Webserver started at http://127.3.220.2:41063/ using document root <none> and password file <none>
I20260809 01:55:16.028946 25040 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:16.029049 25040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:16.032939 25040 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:55:16.035032 25068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:16.036247 25040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:55:16.036384 25040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "3634b4bcc1b24e3d868556cccf946d7b"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "485f25fd351d766434863c3eb09e5b79"
server_key_iv: "6e08593097a67fa8e6b6d772ac74ceb6"
server_key_version: "encryptionkey@0"
I20260809 01:55:16.036911 25040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:16.059036 25040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:16.062038 25040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:16.062252 25040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:16.062848 25040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:16.063666 25040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:16.063736 25040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:16.063803 25040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:16.063850 25040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:16.072911 25040 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:42525
I20260809 01:55:16.072933 25236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:42525 every 8 connection(s)
I20260809 01:55:16.073995 25040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:55:16.082175 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 25040
I20260809 01:55:16.082729 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:39527
--local_ip_for_outbound_sockets=127.3.220.3
--tserver_master_addrs=127.3.220.62:39463
--webserver_port=42167
--webserver_interface=127.3.220.3
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:55:16 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:16.087922 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.075837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:36041 (local address 127.3.220.62:39463)
0809 01:55:16.075974 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.075978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.076976 (+ 998us) server_negotiation.cc:407] Connection header received
0809 01:55:16.077900 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.077903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.077959 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.078075 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:16.079201 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.079729 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.080822 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.081054 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.081142 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.083888 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:16.083895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:16.083901 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:16.083937 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:16.085575 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:16.086272 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:16.086275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:16.086276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:16.086361 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:16.086781 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:16.086783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:16.086784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:16.086952 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:16.087027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.087685 (+ 658us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.087796 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:55:16.089288 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.076188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:36041)
0809 01:55:16.076826 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:16.076866 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:16.077687 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:16.078240 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:16.078247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:16.078606 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:16.079062 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.079075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.079834 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.079837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.080693 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.080702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.081186 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.081188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.081333 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.081980 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:16.082003 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:16.083718 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:16.085700 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:16.085708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:16.085725 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:16.086164 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:16.086456 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:16.086462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:16.086467 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:16.086682 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:16.087060 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:16.087070 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:16.087415 (+ 345us) client_negotiation.cc:769] Sending connection context
0809 01:55:16.087668 (+ 253us) client_negotiation.cc:240] Negotiation successful
0809 01:55:16.088077 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":109,"threads_started":1}
I20260809 01:55:16.090620 25237 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39463
I20260809 01:55:16.090864 25237 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:16.091396 25237 heartbeater.cc:511] Master 127.3.220.62:39463 requested a full tablet report, sending...
I20260809 01:55:16.092489 24766 ts_manager.cc:194] Registered new tserver with Master: 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525)
I20260809 01:55:16.093060 24766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:36041
W20260809 01:55:16.204255 25240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:16.204562 25240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:16.204648 25240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:16.208904 25240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:16.208981 25240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:16.209084 25240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:55:16.214243 25240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:39527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=42167
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:16.215260 25240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:16.216265 25240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:16.226696 25245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:16.226712 25249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:16.226725 25246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:16.227067 25240 server_base.cc:1034] running on GCE node
I20260809 01:55:16.227478 25240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:16.228078 25240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:16.229233 25240 hybrid_clock.cc:648] HybridClock initialized: now 1786240516229199 us; error 47 us; skew 500 ppm
Aug 09 01:55:16 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:16.232152 25240 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:55:16.232999 25240 webserver.cc:460] Webserver started at http://127.3.220.3:42167/ using document root <none> and password file <none>
I20260809 01:55:16.233608 25240 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:16.233702 25240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:16.237500 25240 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:55:16.239738 25262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:16.241150 25240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:55:16.241274 25240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "1c19d7c8623f4b9da3ea6d735e1090e9"
format_stamp: "Formatted at 2026-08-09 01:55:15 on dist-test-slave-gkw7"
server_key: "85666a6eac25ef6d99893a24d2e914ef"
server_key_iv: "263be5e6a963c5fc3047a4dca2931b9b"
server_key_version: "encryptionkey@0"
I20260809 01:55:16.241868 25240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:16.254292 25240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:16.257304 25240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:16.257514 25240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:16.258127 25240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:16.258949 25240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:16.259019 25240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:16.259097 25240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:16.259137 25240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:16.268128 25240 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:39527
I20260809 01:55:16.268157 25429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:39527 every 8 connection(s)
I20260809 01:55:16.269115 25240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:55:16.271194 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 25240
Aug 09 01:55:16 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:16.281162 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.271867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.3.220.62:39463)
0809 01:55:16.272074 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.272079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.272659 (+ 580us) server_negotiation.cc:407] Connection header received
0809 01:55:16.272836 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.272839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.272893 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.272986 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:16.273900 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.274454 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.275653 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.275894 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.275996 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.277689 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:16.277692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:16.277695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:16.277723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:16.279351 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:16.279821 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:16.279823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:16.279825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:16.279899 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:16.280196 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:16.280199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:16.280202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:16.280435 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:16.280658 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.280828 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.280941 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":60,"threads_started":1}
I20260809 01:55:16.283773 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.271089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:46111 (local address 127.3.220.62:39463)
0809 01:55:16.271246 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.271251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.272477 (+ 1226us) server_negotiation.cc:407] Connection header received
0809 01:55:16.273390 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.273396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.273452 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.273587 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:16.274720 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.275483 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.276730 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.277088 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.277216 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.280100 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:16.280102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:16.280103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:16.280119 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:16.281663 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:16.282248 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:16.282254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:16.282258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:16.282347 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:16.282710 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:16.282713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:16.282715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:16.282936 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:16.283034 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.283516 (+ 482us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.283630 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 01:55:16.284570 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.271519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:46111)
0809 01:55:16.272301 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:16.272344 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:16.273191 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:16.273746 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:16.273753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:16.274097 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:16.274592 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.274605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.275594 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.275596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.276607 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.276624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.277236 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.277239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.277403 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.277995 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:16.278018 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:16.279958 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:16.281768 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:16.281773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:16.281786 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:16.282056 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:16.282446 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:16.282449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:16.282451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:16.282597 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:16.283035 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:16.283042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:16.283288 (+ 246us) client_negotiation.cc:769] Sending connection context
0809 01:55:16.283474 (+ 186us) client_negotiation.cc:240] Negotiation successful
0809 01:55:16.283817 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":139,"threads_started":1}
I20260809 01:55:16.285606 25430 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39463
I20260809 01:55:16.285857 25430 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:16.286347 25430 heartbeater.cc:511] Master 127.3.220.62:39463 requested a full tablet report, sending...
I20260809 01:55:16.287340 24766 ts_manager.cc:194] Registered new tserver with Master: 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
I20260809 01:55:16.287883 24766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:46111
I20260809 01:55:16.294624 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:55:16.298756 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.295254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.3.220.62:39463)
0809 01:55:16.295409 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.295414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.295598 (+ 184us) server_negotiation.cc:407] Connection header received
0809 01:55:16.295716 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.295718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.295809 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.295895 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:55:16.296087 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.296662 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.297478 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.297749 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.297831 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.298108 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:55:16.298472 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:55:16.298559 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.298619 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.298660 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 01:55:16.299346 24766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:55:16.302606 24766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:55:16.310477 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfea9909334474aaaebc9754fd2c777 (DEFAULT_TABLE table=test-table [id=2200723a1e2440b79417d00db22e9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:16.311085 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfea9909334474aaaebc9754fd2c777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:16.314108 24877 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Bootstrap starting.
I20260809 01:55:16.314900 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.309123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:42525 (local address 127.0.0.1:44564)
0809 01:55:16.309244 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:16.309261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:16.309341 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:16.310273 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:16.310277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:16.310296 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:16.310397 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.310403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.312278 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.312281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.313202 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.313212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.314393 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.314397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.314547 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.314589 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 01:55:16.314658 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 01:55:16.314745 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":60}
I20260809 01:55:16.314911 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.309379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:39527 (local address 127.0.0.1:51480)
0809 01:55:16.309696 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:16.309709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:16.309796 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:16.310496 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:16.310500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:16.310518 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:16.310606 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.310612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.312049 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.312052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.313338 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.313349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.314393 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.314397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.314598 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.314625 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:55:16.314697 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 01:55:16.314767 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":71,"thread_start_us":96,"threads_started":1}
I20260809 01:55:16.314965 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.309660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.3.220.3:39527)
0809 01:55:16.310041 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.310052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.310073 (+ 21us) server_negotiation.cc:407] Connection header received
0809 01:55:16.310143 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.310148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.310339 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.310758 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:16.310901 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.311938 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.313467 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.314242 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.314522 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.314679 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.314733 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.314823 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":107,"threads_started":1}
I20260809 01:55:16.315083 24877 tablet_bootstrap.cc:654] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:16.315616 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.309321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.3.220.2:42525)
0809 01:55:16.309692 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.309706 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.309736 (+ 30us) server_negotiation.cc:407] Connection header received
0809 01:55:16.309872 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.309880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.310112 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.310275 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:16.310498 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.312134 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.313326 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.314242 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.315160 (+ 918us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.315341 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.315393 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.315478 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260809 01:55:16.316956 24877 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: No bootstrap required, opened a new log
I20260809 01:55:16.317049 24877 ts_tablet_manager.cc:1397] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260809 01:55:16.317487 24877 raft_consensus.cc:348] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.317646 24877 raft_consensus.cc:374] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:16.317694 24877 raft_consensus.cc:729] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ef5cf560094a71ba808c7992215b22, State: Initialized, Role: FOLLOWER
I20260809 01:55:16.317858 24877 consensus_queue.cc:260] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.318578 24877 ts_tablet_manager.cc:1428] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 01:55:16.318238 25338 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfea9909334474aaaebc9754fd2c777 (DEFAULT_TABLE table=test-table [id=2200723a1e2440b79417d00db22e9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:16.318229 25135 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfea9909334474aaaebc9754fd2c777 (DEFAULT_TABLE table=test-table [id=2200723a1e2440b79417d00db22e9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:16.319286 25135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfea9909334474aaaebc9754fd2c777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:16.319312 25338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfea9909334474aaaebc9754fd2c777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:16.325947 25448 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: Bootstrap starting.
I20260809 01:55:16.325947 25449 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: Bootstrap starting.
I20260809 01:55:16.327847 25448 tablet_bootstrap.cc:654] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:16.328230 25449 tablet_bootstrap.cc:654] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:16.329110 25448 log.cc:824] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:16.329146 25449 log.cc:824] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:16.330884 25449 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: No bootstrap required, opened a new log
I20260809 01:55:16.330951 25448 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: No bootstrap required, opened a new log
I20260809 01:55:16.331111 25449 ts_tablet_manager.cc:1397] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:55:16.331141 25448 ts_tablet_manager.cc:1397] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:55:16.333847 25449 raft_consensus.cc:348] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.333849 25448 raft_consensus.cc:348] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.334086 25449 raft_consensus.cc:374] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:16.334092 25448 raft_consensus.cc:374] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:16.334141 25449 raft_consensus.cc:729] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3634b4bcc1b24e3d868556cccf946d7b, State: Initialized, Role: FOLLOWER
I20260809 01:55:16.334146 25448 raft_consensus.cc:729] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c19d7c8623f4b9da3ea6d735e1090e9, State: Initialized, Role: FOLLOWER
I20260809 01:55:16.334632 25448 consensus_queue.cc:260] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.334618 25449 consensus_queue.cc:260] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.335285 25237 heartbeater.cc:503] Master 127.3.220.62:39463 was elected leader, sending a full tablet report...
I20260809 01:55:16.335306 25430 heartbeater.cc:503] Master 127.3.220.62:39463 was elected leader, sending a full tablet report...
I20260809 01:55:16.335640 25449 ts_tablet_manager.cc:1428] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 01:55:16.335641 25448 ts_tablet_manager.cc:1428] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260809 01:55:16.371428 25039 tablet.cc:2369] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:55:16.472609 25454 raft_consensus.cc:482] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:16.472898 25454 raft_consensus.cc:504] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.474490 25454 leader_election.cc:288] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 92ef5cf560094a71ba808c7992215b22 (127.3.220.1:46701)
I20260809 01:55:16.478677 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.474732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:42525 (local address 127.3.220.3:38337)
0809 01:55:16.474868 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:16.474885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:16.474987 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:16.475408 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:16.475411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:16.475434 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:16.475563 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.475570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.476485 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.476488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.477425 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.477434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.478224 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.478228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.478340 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.478368 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 01:55:16.478449 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 01:55:16.478524 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 01:55:16.478685 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.474865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:38337 (local address 127.3.220.2:42525)
0809 01:55:16.475017 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.475022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.475039 (+ 17us) server_negotiation.cc:407] Connection header received
0809 01:55:16.475102 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.475104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.475159 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.475256 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:16.475678 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.476358 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.477583 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.478097 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.478186 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.478244 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.478478 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.478555 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:55:16.479053 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.475030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46701 (local address 127.3.220.3:48741)
0809 01:55:16.475404 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:16.475416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:16.475493 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:16.475837 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:16.475839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:16.475857 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:16.475978 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.475983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.476837 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.476839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.477798 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:16.477809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.478613 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.478617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:16.478726 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.478753 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:55:16.478842 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 01:55:16.478921 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":164,"threads_started":1}
I20260809 01:55:16.479331 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.475159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:48741 (local address 127.3.220.1:46701)
0809 01:55:16.475480 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.475484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.475497 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:55:16.475575 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.475578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.475651 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.475829 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:16.476090 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.476719 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.477924 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.478493 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.478595 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.478761 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.478994 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.479072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260809 01:55:16.479385 25168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfea9909334474aaaebc9754fd2c777" candidate_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3634b4bcc1b24e3d868556cccf946d7b" is_pre_election: true
I20260809 01:55:16.479743 25168 raft_consensus.cc:2454] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c19d7c8623f4b9da3ea6d735e1090e9 in term 0.
I20260809 01:55:16.479868 24971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfea9909334474aaaebc9754fd2c777" candidate_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ef5cf560094a71ba808c7992215b22" is_pre_election: true
I20260809 01:55:16.480139 24971 raft_consensus.cc:2454] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c19d7c8623f4b9da3ea6d735e1090e9 in term 0.
I20260809 01:55:16.480252 25263 leader_election.cc:302] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c19d7c8623f4b9da3ea6d735e1090e9, 3634b4bcc1b24e3d868556cccf946d7b; no voters:
I20260809 01:55:16.480556 25454 raft_consensus.cc:2781] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:55:16.480654 25454 raft_consensus.cc:482] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:55:16.480717 25454 raft_consensus.cc:3037] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:16.481897 25454 raft_consensus.cc:504] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.482234 25454 leader_election.cc:288] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [CANDIDATE]: Term 1 election: Requested vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 92ef5cf560094a71ba808c7992215b22 (127.3.220.1:46701)
I20260809 01:55:16.482615 25168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfea9909334474aaaebc9754fd2c777" candidate_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3634b4bcc1b24e3d868556cccf946d7b"
I20260809 01:55:16.482662 24971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfea9909334474aaaebc9754fd2c777" candidate_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ef5cf560094a71ba808c7992215b22"
I20260809 01:55:16.482785 24971 raft_consensus.cc:3037] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:16.482784 25168 raft_consensus.cc:3037] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:16.483776 24971 raft_consensus.cc:2454] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c19d7c8623f4b9da3ea6d735e1090e9 in term 1.
I20260809 01:55:16.483924 25168 raft_consensus.cc:2454] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c19d7c8623f4b9da3ea6d735e1090e9 in term 1.
I20260809 01:55:16.484063 25263 leader_election.cc:302] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c19d7c8623f4b9da3ea6d735e1090e9, 92ef5cf560094a71ba808c7992215b22; no voters:
I20260809 01:55:16.484294 25454 raft_consensus.cc:2781] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:16.484551 25454 raft_consensus.cc:686] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 LEADER]: Becoming Leader. State: Replica: 1c19d7c8623f4b9da3ea6d735e1090e9, State: Running, Role: LEADER
I20260809 01:55:16.484912 25454 consensus_queue.cc:237] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:16.488155 24766 catalog_manager.cc:5495] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3). New cstate: current_term: 1 leader_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } health_report { overall_health: UNKNOWN } } }
I20260809 01:55:16.506953 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.503389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.3.220.3:39527)
0809 01:55:16.503531 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:16.503535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:16.503549 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:55:16.503710 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:16.503712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:16.503769 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:16.503878 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:55:16.504109 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.504629 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.505464 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:16.505716 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:16.505825 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:16.506124 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:55:16.506532 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:55:16.506662 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:16.506733 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 01:55:16.506790 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:55:16.512804 25168 raft_consensus.cc:1261] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 1 FOLLOWER]: Refusing update from remote peer 1c19d7c8623f4b9da3ea6d735e1090e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:16.512800 24971 raft_consensus.cc:1261] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Refusing update from remote peer 1c19d7c8623f4b9da3ea6d735e1090e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:16.513599 25454 consensus_queue.cc:1035] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:16.513793 25457 consensus_queue.cc:1035] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:16.521862 25460 mvcc.cc:205] Tried to move back new op lower bound from 7316441155629137920 to 7316441155525087232. Current Snapshot: MvccSnapshot[applied={T|T < 7316441155629137920}]
I20260809 01:55:16.522287 25464 mvcc.cc:205] Tried to move back new op lower bound from 7316441155629137920 to 7316441155525087232. Current Snapshot: MvccSnapshot[applied={T|T < 7316441155629137920}]
I20260809 01:55:16.530193 24766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:16.530349 24766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:16.532796 24766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Sending DeleteTablet for 3 replicas of tablet 4bfea9909334474aaaebc9754fd2c777
I20260809 01:55:16.533515 24941 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfea9909334474aaaebc9754fd2c777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:16 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:58550
I20260809 01:55:16.533627 25338 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfea9909334474aaaebc9754fd2c777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:16 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:51480
I20260809 01:55:16.533916 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 24819
I20260809 01:55:16.533957 25135 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfea9909334474aaaebc9754fd2c777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:16 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:44564
I20260809 01:55:16.534273 25475 tablet_replica.cc:330] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: stopping tablet replica
I20260809 01:55:16.536504 25475 raft_consensus.cc:2229] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:55:16.537160 25475 raft_consensus.cc:2258] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:16.541348 25475 ts_tablet_manager.cc:1905] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:16.545183 25476 tablet_replica.cc:330] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: stopping tablet replica
I20260809 01:55:16.545572 25476 raft_consensus.cc:2229] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:16.545953 25476 raft_consensus.cc:2258] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:16.546031 25475 ts_tablet_manager.cc:1918] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:16.546154 25475 log.cc:1196] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/4bfea9909334474aaaebc9754fd2c777
I20260809 01:55:16.546550 25475 ts_tablet_manager.cc:1939] T 4bfea9909334474aaaebc9754fd2c777 P 1c19d7c8623f4b9da3ea6d735e1090e9: Deleting consensus metadata
I20260809 01:55:16.548098 24745 catalog_manager.cc:4841] TS 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527): tablet 4bfea9909334474aaaebc9754fd2c777 (table test-table [id=2200723a1e2440b79417d00db22e9ff3]) successfully deleted
I20260809 01:55:16.550598 25476 ts_tablet_manager.cc:1905] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:16.552959 25476 ts_tablet_manager.cc:1918] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:16.553093 25476 log.cc:1196] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/4bfea9909334474aaaebc9754fd2c777
I20260809 01:55:16.553475 25476 ts_tablet_manager.cc:1939] T 4bfea9909334474aaaebc9754fd2c777 P 3634b4bcc1b24e3d868556cccf946d7b: Deleting consensus metadata
I20260809 01:55:16.554754 24744 catalog_manager.cc:4841] TS 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525): tablet 4bfea9909334474aaaebc9754fd2c777 (table test-table [id=2200723a1e2440b79417d00db22e9ff3]) successfully deleted
W20260809 01:55:16.556370 24744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:46701 (error 108)
I20260809 01:55:16.557060 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 25040
I20260809 01:55:16.560563 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:16.557334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:46701 (local address 127.0.0.1:45070)
0809 01:55:16.560361 (+ 3027us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:16.560452 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:46701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2908}
W20260809 01:55:16.565935 24744 catalog_manager.cc:4584] TS 92ef5cf560094a71ba808c7992215b22 (127.3.220.1:46701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bfea9909334474aaaebc9754fd2c777: Network error: Client connection negotiation failed: client connection to 127.3.220.1:46701: connect: Connection refused (error 111)
I20260809 01:55:16.575248 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 25240
I20260809 01:55:16.588306 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 24720
I20260809 01:55:16.600906 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39463
--webserver_interface=127.3.220.62
--webserver_port=41853
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:39463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:16.714397 25479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:16.714671 25479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:16.714711 25479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:16.718745 25479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:16.718807 25479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:16.718827 25479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:16.718844 25479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:16.718859 25479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:16.724208 25479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:39463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=41853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:16.725291 25479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:16.726416 25479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:16.735623 25488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:16.735675 25479 server_base.cc:1034] running on GCE node
W20260809 01:55:16.735715 25489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:16.735859 25494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:16.736433 25479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:16.737632 25479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:16.738847 25479 hybrid_clock.cc:648] HybridClock initialized: now 1786240516738827 us; error 32 us; skew 500 ppm
Aug 09 01:55:16 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:16.741601 25479 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:16.742436 25479 webserver.cc:460] Webserver started at http://127.3.220.62:41853/ using document root <none> and password file <none>
I20260809 01:55:16.742962 25479 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:16.743021 25479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:16.747172 25479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:55:16.749477 25503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:16.750787 25479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:55:16.750934 25479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "00a661d5bfe84ee3821337014243f5ec"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "9ed7b90d4cfa7c359816ae4154fc681f"
server_key_iv: "d1323df3fcfc565965166886fecfb468"
server_key_version: "encryptionkey@0"
I20260809 01:55:16.751497 25479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:16.787221 25479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:16.790597 25479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:16.790829 25479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:16.797762 25479 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:39463
I20260809 01:55:16.798107 25579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:39463 every 8 connection(s)
I20260809 01:55:16.798907 25479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:16.803200 25580 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:55:16.804805 25580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Bootstrap starting.
I20260809 01:55:16.808522 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 25479
I20260809 01:55:16.808956 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:46701
--local_ip_for_outbound_sockets=127.3.220.1
--tserver_master_addrs=127.3.220.62:39463
--webserver_port=33847
--webserver_interface=127.3.220.1
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:16.822885 25580 log.cc:824] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:16.834141 25580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:55:16.834545 25580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: Bootstrap complete.
I20260809 01:55:16.838742 25580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:16.839195 25580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a661d5bfe84ee3821337014243f5ec, State: Initialized, Role: FOLLOWER
I20260809 01:55:16.839746 25580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:16.839865 25580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:55:16.839949 25580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:55:16.840058 25580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 2 FOLLOWER]: Advancing to term 3
I20260809 01:55:16.842056 25580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:16.842391 25580 leader_election.cc:302] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a661d5bfe84ee3821337014243f5ec; no voters:
I20260809 01:55:16.842703 25580 leader_election.cc:288] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 01:55:16.842834 25586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 3 FOLLOWER]: Leader election won for term 3
I20260809 01:55:16.843096 25586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [term 3 LEADER]: Becoming Leader. State: Replica: 00a661d5bfe84ee3821337014243f5ec, State: Running, Role: LEADER
I20260809 01:55:16.843412 25586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } }
I20260809 01:55:16.843938 25580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:16.845012 25587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "00a661d5bfe84ee3821337014243f5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } } }
I20260809 01:55:16.845130 25587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:16.844390 25586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a661d5bfe84ee3821337014243f5ec. Latest consensus state: current_term: 3 leader_uuid: "00a661d5bfe84ee3821337014243f5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a661d5bfe84ee3821337014243f5ec" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39463 } } }
I20260809 01:55:16.845273 25586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:16.848764 25607 catalog_manager.cc:1259] Loaded cluster ID: b550f8ea33734f6c8a53dfb2ff65a3c8
I20260809 01:55:16.848851 25607 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: loading cluster ID for follower catalog manager: success
W20260809 01:55:16.849877 25607 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 00a661d5bfe84ee3821337014243f5ec: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260809 01:55:16.849952 25607 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260809 01:55:16.850009 25602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:16.852113 25602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:16.852571 25602 catalog_manager.cc:1259] Loaded cluster ID: b550f8ea33734f6c8a53dfb2ff65a3c8
I20260809 01:55:16.852644 25602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 01:55:16.853174 25602 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f290eb8d9ad google::LogMessage::Fail() at ??:0
@ 0x7f290eb9088a google::LogMessage::SendToLog() at ??:0
@ 0x7f290eb8d4de google::LogMessage::Flush() at ??:0
@ 0x7f290eb8ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f29194ccfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f29194cdd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f29194ca579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f291950e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2910ddf3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2910ddfc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2910de13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2910dd0512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f29124146db start_thread at ??:0
@ 0x7f290d68b61f clone at ??:0
W20260809 01:55:16.955302 25583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:16.955575 25583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:16.955621 25583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:16.959820 25583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:16.959890 25583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:16.959952 25583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:55:16.965564 25583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:46701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=33847
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:16.966694 25583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:16.967754 25583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:16.978844 25645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:16.978844 25642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:16.978861 25643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:16.979662 25583 server_base.cc:1034] running on GCE node
I20260809 01:55:16.980182 25583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:16.980880 25583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:16.982065 25583 hybrid_clock.cc:648] HybridClock initialized: now 1786240516982034 us; error 49 us; skew 500 ppm
Aug 09 01:55:16 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:16.985006 25583 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:55:16.986088 25583 webserver.cc:460] Webserver started at http://127.3.220.1:33847/ using document root <none> and password file <none>
I20260809 01:55:16.986682 25583 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:16.986794 25583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:16.990863 25583 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260809 01:55:16.993098 25653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:16.994482 25583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:55:16.994601 25583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "92ef5cf560094a71ba808c7992215b22"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "6f178e85f4a168ffe903c81f58cb405e"
server_key_iv: "91f48bfcdf649da03a6320bc49446b05"
server_key_version: "encryptionkey@0"
I20260809 01:55:16.995175 25583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:17.010282 25583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:17.014315 25583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:17.014533 25583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:17.015290 25583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:17.016755 25665 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 01:55:17.021003 25583 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 01:55:17.021082 25583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260809 01:55:17.021157 25583 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 01:55:17.023404 25583 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 01:55:17.023466 25583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260809 01:55:17.023833 25665 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Bootstrap starting.
I20260809 01:55:17.034446 25583 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:46701
I20260809 01:55:17.034471 25811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:46701 every 8 connection(s)
I20260809 01:55:17.035704 25583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:55:17.037845 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 25583
I20260809 01:55:17.038446 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:42525
--local_ip_for_outbound_sockets=127.3.220.2
--tserver_master_addrs=127.3.220.62:39463
--webserver_port=41063
--webserver_interface=127.3.220.2
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:17.045316 25665 log.cc:824] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:17.056208 25665 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:55:17.056674 25665 tablet_bootstrap.cc:492] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Bootstrap complete.
I20260809 01:55:17.058221 25665 ts_tablet_manager.cc:1397] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.007s
I20260809 01:55:17.061435 25665 raft_consensus.cc:348] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:17.062057 25665 raft_consensus.cc:729] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ef5cf560094a71ba808c7992215b22, State: Initialized, Role: FOLLOWER
I20260809 01:55:17.062578 25665 consensus_queue.cc:260] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
W20260809 01:55:17.063516 25816 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:55:17.038148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:58847)
0809 01:55:17.038706 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.038761 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:17.039910 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:17.062574 (+ 22664us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":175,"threads_started":1}
I20260809 01:55:17.063619 25665 ts_tablet_manager.cc:1428] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260809 01:55:17.063969 25814 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:39463 (0 consecutive failures): Network error: Failed to ping master at 127.3.220.62:39463: Client connection negotiation failed: client connection to 127.3.220.62:39463: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 01:55:17.064672 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:17.064448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:46943)
0809 01:55:17.064560 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.064619 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 01:55:17.065274 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:17.065077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:54043)
0809 01:55:17.065180 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.065224 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 01:55:17.065450 25814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 01:55:17.164508 25819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:17.164755 25819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:17.164798 25819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:17.168767 25819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:17.168825 25819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:17.168915 25819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:55:17.174391 25819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:42525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=41063
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:17.175400 25819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:17.176414 25819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:17.186519 25837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:17.186519 25835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:17.186707 25819 server_base.cc:1034] running on GCE node
W20260809 01:55:17.186637 25833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:17.187290 25819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:17.188035 25819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:17.189239 25819 hybrid_clock.cc:648] HybridClock initialized: now 1786240517189204 us; error 49 us; skew 500 ppm
Aug 09 01:55:17 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:17.192008 25819 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:55:17.192914 25819 webserver.cc:460] Webserver started at http://127.3.220.2:41063/ using document root <none> and password file <none>
I20260809 01:55:17.193495 25819 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:17.193729 25819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:17.197907 25819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260809 01:55:17.200109 25847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:17.201315 25819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:55:17.201447 25819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "3634b4bcc1b24e3d868556cccf946d7b"
format_stamp: "Formatted at 2026-08-09 01:55:14 on dist-test-slave-gkw7"
server_key: "485f25fd351d766434863c3eb09e5b79"
server_key_iv: "6e08593097a67fa8e6b6d772ac74ceb6"
server_key_version: "encryptionkey@0"
I20260809 01:55:17.202101 25819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:17.217855 25819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:17.220822 25819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:17.221026 25819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:17.221693 25819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:17.222576 25819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:17.222659 25819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:17.222752 25819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:17.222803 25819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:17.232456 25819 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:42525
I20260809 01:55:17.232483 26008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:42525 every 8 connection(s)
I20260809 01:55:17.233510 25819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:55:17.237079 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:17.235647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:48175)
0809 01:55:17.236299 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.236414 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":116,"threads_started":1}
W20260809 01:55:17.237576 26009 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:39463 (0 consecutive failures): Network error: Failed to ping master at 127.3.220.62:39463: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
I20260809 01:55:17.238134 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:17.237939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:46733)
0809 01:55:17.238037 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.238084 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 01:55:17.238824 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:17.238633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:34515)
0809 01:55:17.238740 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.238780 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 01:55:17.239029 26009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 01:55:17.241125 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 25819
I20260809 01:55:17.241645 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:39527
--local_ip_for_outbound_sockets=127.3.220.3
--tserver_master_addrs=127.3.220.62:39463
--webserver_port=42167
--webserver_interface=127.3.220.3
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:17.358023 26013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:17.358393 26013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:17.358471 26013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:17.362555 26013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:17.362607 26013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:17.362664 26013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:55:17.367847 26013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:39527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=42167
--tserver_master_addrs=127.3.220.62:39463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:17.368849 26013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:17.369886 26013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:17.379207 26020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:17.379240 26019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:17.379494 26013 server_base.cc:1034] running on GCE node
W20260809 01:55:17.379225 26023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:17.379998 26013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:17.380573 26013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:17.381738 26013 hybrid_clock.cc:648] HybridClock initialized: now 1786240517381722 us; error 27 us; skew 500 ppm
Aug 09 01:55:17 dist-test-slave-gkw7 krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:17.384690 26013 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:55:17.385617 26013 webserver.cc:460] Webserver started at http://127.3.220.3:42167/ using document root <none> and password file <none>
I20260809 01:55:17.386246 26013 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:17.386319 26013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:17.390476 26013 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:55:17.392573 26031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:17.393971 26013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:55:17.394117 26013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "1c19d7c8623f4b9da3ea6d735e1090e9"
format_stamp: "Formatted at 2026-08-09 01:55:15 on dist-test-slave-gkw7"
server_key: "85666a6eac25ef6d99893a24d2e914ef"
server_key_iv: "263be5e6a963c5fc3047a4dca2931b9b"
server_key_version: "encryptionkey@0"
I20260809 01:55:17.394685 26013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:17.409830 26013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:17.413061 26013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:17.413278 26013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:17.413929 26013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:17.414747 26013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:17.414820 26013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:17.414892 26013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:17.414933 26013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:17.423961 26013 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:39527
I20260809 01:55:17.423996 26184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:39527 every 8 connection(s)
I20260809 01:55:17.424968 26013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:55:17.428539 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:17.427082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:40311)
0809 01:55:17.427730 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.427845 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":116,"threads_started":1}
W20260809 01:55:17.429021 26185 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:39463 (0 consecutive failures): Network error: Failed to ping master at 127.3.220.62:39463: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
I20260809 01:55:17.429028 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 26013
I20260809 01:55:17.429733 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:17.429455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:43421)
0809 01:55:17.429609 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.429665 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 01:55:17.430440 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:17.430251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:36489)
0809 01:55:17.430353 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:17.430393 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 01:55:17.430649 26185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 01:55:18.066560 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:18.066142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:53295)
0809 01:55:18.066386 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:18.066444 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260809 01:55:18.240022 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:18.239571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:47125)
0809 01:55:18.239821 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:18.239877 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260809 01:55:18.431545 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:18.431164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:52535)
0809 01:55:18.431404 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:18.431455 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260809 01:55:18.441661 26195 raft_consensus.cc:482] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:18.441897 26195 raft_consensus.cc:504] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:18.443120 26195 leader_election.cc:288] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
Aug 09 01:55:18 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.2@KRBTEST.COM
Aug 09 01:55:18 dist-test-slave-gkw7 krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.3@KRBTEST.COM
I20260809 01:55:18.463217 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:18.443540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:48035 (local address 127.3.220.2:42525)
0809 01:55:18.443974 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:18.443992 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:18.444023 (+ 31us) server_negotiation.cc:407] Connection header received
0809 01:55:18.444638 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:18.444643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:18.444965 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:18.445270 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:18.446565 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:18.447571 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:18.449526 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:18.450249 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:18.450431 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:18.458488 (+ 8057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:18.458500 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:18.458520 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:18.458561 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:18.460453 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:18.461070 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:18.461077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:18.461082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:18.461218 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:18.461627 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:18.461632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:18.461635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:18.462058 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:18.462191 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:18.462763 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 01:55:18.463040 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":128,"threads_started":1}
I20260809 01:55:18.463205 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:18.443360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:39527 (local address 127.3.220.1:58445)
0809 01:55:18.443715 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:18.443730 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:18.443832 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:18.445313 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:18.445319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:18.445926 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:18.446393 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:18.446407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:18.447770 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:18.447774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:18.449405 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:18.449411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:18.450384 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:18.450389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:18.450669 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:18.456058 (+ 5389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:18.456101 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:18.458522 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:18.460727 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:18.460730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:18.460732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:18.461003 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:18.461340 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:18.461343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:18.461345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:18.461509 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:18.462197 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:18.462201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:18.462432 (+ 231us) client_negotiation.cc:769] Sending connection context
0809 01:55:18.462599 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 01:55:18.462983 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":342,"spinlock_wait_cycles":55296,"thread_start_us":82,"threads_started":1}
I20260809 01:55:18.463258 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:18.443321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:42525 (local address 127.3.220.1:48035)
0809 01:55:18.443526 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:18.443542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:18.443654 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:18.445318 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:18.445322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:18.445925 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:18.446393 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:18.446406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:18.447731 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:18.447737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:18.449397 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:18.449411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:18.450384 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:18.450389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:18.450660 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:18.456058 (+ 5398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:18.456091 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:18.458220 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:18.460588 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:18.460597 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:18.460611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:18.460941 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:18.461328 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:18.461333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:18.461335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:18.461496 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:18.462183 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:18.462190 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:18.462435 (+ 245us) client_negotiation.cc:769] Sending connection context
0809 01:55:18.462609 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 01:55:18.463071 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":444,"spinlock_wait_cycles":27776}
I20260809 01:55:18.463276 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:18.443534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:58445 (local address 127.3.220.3:39527)
0809 01:55:18.443922 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:18.443945 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:18.443983 (+ 38us) server_negotiation.cc:407] Connection header received
0809 01:55:18.444649 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:18.444654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:18.444966 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:18.445282 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:18.446583 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:18.447575 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:18.449543 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:18.450249 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:18.450432 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:18.458738 (+ 8306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:18.458744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:18.458758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:18.458792 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:18.460602 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:18.461113 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:18.461123 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:18.461128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:18.461236 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:18.461642 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:18.461646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:18.461648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:18.462060 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:18.462219 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:18.462767 (+ 548us) server_negotiation.cc:299] Negotiation successful
0809 01:55:18.463064 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
W20260809 01:55:18.466143 25657 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
W20260809 01:55:18.466480 25654 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
I20260809 01:55:18.466591 25654 leader_election.cc:302] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ef5cf560094a71ba808c7992215b22; no voters: 1c19d7c8623f4b9da3ea6d735e1090e9, 3634b4bcc1b24e3d868556cccf946d7b
I20260809 01:55:18.466835 26195 raft_consensus.cc:2726] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:55:19.067900 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:19.067444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:34435)
0809 01:55:19.067729 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:19.067793 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260809 01:55:19.241181 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:19.240781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:51913)
0809 01:55:19.241024 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:19.241074 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260809 01:55:19.432859 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:19.432396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:39919)
0809 01:55:19.432673 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:19.432749 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260809 01:55:20.069275 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:20.068801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:40741)
0809 01:55:20.069069 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:20.069160 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260809 01:55:20.242415 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:20.241979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:37065)
0809 01:55:20.242254 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:20.242310 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
I20260809 01:55:20.434114 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:20.433676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:33447)
0809 01:55:20.433948 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:20.434006 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260809 01:55:20.710350 26209 raft_consensus.cc:482] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:20.710528 26209 raft_consensus.cc:504] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:20.711010 26209 leader_election.cc:288] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
W20260809 01:55:20.711807 25657 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
W20260809 01:55:20.711944 25654 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
I20260809 01:55:20.712038 25654 leader_election.cc:302] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ef5cf560094a71ba808c7992215b22; no voters: 1c19d7c8623f4b9da3ea6d735e1090e9, 3634b4bcc1b24e3d868556cccf946d7b
I20260809 01:55:20.712210 26209 raft_consensus.cc:2726] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:55:21.070572 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:21.070151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:56021)
0809 01:55:21.070416 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:21.070472 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260809 01:55:21.243745 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:21.243232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:60115)
0809 01:55:21.243537 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:21.243603 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260809 01:55:21.435382 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:21.434891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:38683)
0809 01:55:21.435195 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:21.435253 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":118,"threads_started":1}
I20260809 01:55:22.072225 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:22.071636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:58375)
0809 01:55:22.072000 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:22.072078 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":142,"threads_started":1}
I20260809 01:55:22.244933 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:22.244492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:40505)
0809 01:55:22.244781 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:22.244840 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260809 01:55:22.389621 26219 raft_consensus.cc:482] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:22.389783 26219 raft_consensus.cc:504] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:22.390216 26219 leader_election.cc:288] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
W20260809 01:55:22.390974 25654 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
W20260809 01:55:22.391086 25657 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
I20260809 01:55:22.391191 25657 leader_election.cc:302] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ef5cf560094a71ba808c7992215b22; no voters: 1c19d7c8623f4b9da3ea6d735e1090e9, 3634b4bcc1b24e3d868556cccf946d7b
I20260809 01:55:22.391357 26219 raft_consensus.cc:2726] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:55:22.436772 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:22.436274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:50109)
0809 01:55:22.436605 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:22.436676 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260809 01:55:23.073836 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:23.073100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:51627)
0809 01:55:23.073407 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:23.073466 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260809 01:55:23.246207 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:23.245779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:37729)
0809 01:55:23.246048 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:23.246106 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260809 01:55:23.438189 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:23.437747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:59061)
0809 01:55:23.438023 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:23.438087 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
I20260809 01:55:24.075050 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:24.074623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:41609)
0809 01:55:24.074892 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:24.074950 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260809 01:55:24.247501 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:24.246984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:51139)
0809 01:55:24.247346 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:24.247407 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":75,"threads_started":1}
I20260809 01:55:24.439819 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:24.439214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:44107)
0809 01:55:24.439610 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:24.439672 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":118,"threads_started":1}
I20260809 01:55:24.831909 26230 raft_consensus.cc:482] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:24.832096 26230 raft_consensus.cc:504] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:24.832600 26230 leader_election.cc:288] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
W20260809 01:55:24.833375 25654 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
W20260809 01:55:24.833465 25657 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
I20260809 01:55:24.833576 25657 leader_election.cc:302] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ef5cf560094a71ba808c7992215b22; no voters: 1c19d7c8623f4b9da3ea6d735e1090e9, 3634b4bcc1b24e3d868556cccf946d7b
I20260809 01:55:24.833761 26230 raft_consensus.cc:2726] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:55:25.076397 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:25.075904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:53075)
0809 01:55:25.076230 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:25.076283 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
I20260809 01:55:25.248863 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:25.248354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:32957)
0809 01:55:25.248697 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:25.248768 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":81,"threads_started":1}
I20260809 01:55:25.441082 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:25.440674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:44939)
0809 01:55:25.440929 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:25.440988 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260809 01:55:26.077749 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:26.077178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:45251)
0809 01:55:26.077482 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:26.077625 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260809 01:55:26.250108 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:26.249656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:51993)
0809 01:55:26.249935 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:26.249997 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
I20260809 01:55:26.442612 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:26.442028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:52405)
0809 01:55:26.442416 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:26.442492 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":106,"threads_started":1}
I20260809 01:55:27.079201 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:27.078715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:40419)
0809 01:55:27.079006 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:27.079093 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
I20260809 01:55:27.251371 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:27.250916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:37707)
0809 01:55:27.251196 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:27.251255 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260809 01:55:27.443884 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:27.443470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:47149)
0809 01:55:27.443722 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:27.443777 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260809 01:55:28.080484 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:28.080034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:33097)
0809 01:55:28.080323 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:28.080385 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260809 01:55:28.252631 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:28.252151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:40725)
0809 01:55:28.252430 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:28.252514 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260809 01:55:28.445202 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:28.444753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:54973)
0809 01:55:28.445039 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:28.445101 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
I20260809 01:55:28.574936 26250 raft_consensus.cc:482] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:28.575075 26250 raft_consensus.cc:504] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3634b4bcc1b24e3d868556cccf946d7b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42525 } } peers { permanent_uuid: "1c19d7c8623f4b9da3ea6d735e1090e9" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 39527 } } peers { permanent_uuid: "92ef5cf560094a71ba808c7992215b22" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 46701 } }
I20260809 01:55:28.575614 26250 leader_election.cc:288] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525), 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527)
W20260809 01:55:28.576545 25657 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c19d7c8623f4b9da3ea6d735e1090e9 (127.3.220.3:39527): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
W20260809 01:55:28.576719 25654 leader_election.cc:341] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3634b4bcc1b24e3d868556cccf946d7b (127.3.220.2:42525): Not found: Tablet not found: 4bfea9909334474aaaebc9754fd2c777
I20260809 01:55:28.576805 25654 leader_election.cc:302] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ef5cf560094a71ba808c7992215b22; no voters: 1c19d7c8623f4b9da3ea6d735e1090e9, 3634b4bcc1b24e3d868556cccf946d7b
I20260809 01:55:28.576969 26250 raft_consensus.cc:2726] T 4bfea9909334474aaaebc9754fd2c777 P 92ef5cf560094a71ba808c7992215b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:55:29.081671 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:29.081213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:36385)
0809 01:55:29.081469 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:29.081521 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
I20260809 01:55:29.254058 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:29.253599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:39989)
0809 01:55:29.253893 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:29.253962 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20260809 01:55:29.446455 26257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:29.446031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:55889)
0809 01:55:29.446299 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:29.446359 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260809 01:55:30.082939 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:30.082465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:52261)
0809 01:55:30.082770 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:30.082836 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260809 01:55:30.255465 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:30.254973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:56043)
0809 01:55:30.255295 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:30.255368 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":89,"threads_started":1}
I20260809 01:55:30.447705 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:30.447276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:54099)
0809 01:55:30.447534 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:30.447590 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260809 01:55:31.084306 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:31.083796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:40751)
0809 01:55:31.084123 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:31.084188 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":79,"threads_started":1}
I20260809 01:55:31.256986 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:31.256386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:49391)
0809 01:55:31.256747 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:31.256849 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":161,"threads_started":1}
I20260809 01:55:31.448915 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:31.448495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:60577)
0809 01:55:31.448747 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:31.448816 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260809 01:55:32.085911 26264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:32.085364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.1:43287)
0809 01:55:32.085729 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:32.085800 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":77,"threads_started":1}
I20260809 01:55:32.258433 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:32.257995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.2:53575)
0809 01:55:32.258275 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:32.258335 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260809 01:55:32.438781 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 25583
I20260809 01:55:32.450414 26266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:32.449995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39463 (local address 127.3.220.3:43847)
0809 01:55:32.450247 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:32.450319 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260809 01:55:32.454425 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 25819
I20260809 01:55:32.465611 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 26013
2026-08-09T01:55:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19553 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:55:32 dist-test-slave-gkw7 krb5kdc[26271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:55:32 dist-test-slave-gkw7 krb5kdc[26271](info): set up 2 sockets
Aug 09 01:55:32 dist-test-slave-gkw7 krb5kdc[26271](info): commencing operation
krb5kdc: starting...
W20260809 01:55:33.717478 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:55:33 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:55:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:55:33.845376 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35731
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:33.956413 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:33.956723 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:33.956797 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:33.960769 26287 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 01:55:33.960865 26287 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 01:55:33.960893 26287 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 01:55:33.960917 26287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:33.960958 26287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:33.961001 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:33.961059 26287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:33.961098 26287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:33.966336 26287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:33.967413 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:33.968457 26287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:33.977103 26298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:33.977085 26300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:33.977116 26297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:33.977263 26287 server_base.cc:1034] running on GCE node
I20260809 01:55:33.978020 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:33.979122 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:33.980365 26287 hybrid_clock.cc:648] HybridClock initialized: now 1786240533980339 us; error 49 us; skew 500 ppm
Aug 09 01:55:33 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240533, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:33.983362 26287 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:33.984326 26287 webserver.cc:460] Webserver started at http://127.3.220.62:35477/ using document root <none> and password file <none>
I20260809 01:55:33.984943 26287 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:33.985049 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:33.985308 26287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:33.987519 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "13b011436ec644288b1ccac55b6c19f5"
format_stamp: "Formatted at 2026-08-09 01:55:33 on dist-test-slave-gkw7"
server_key: "f3e12311f65e800d7821defee66b392b"
server_key_iv: "c52711d4f0e123181a0973e63bf84097"
server_key_version: "encryptionkey@0"
I20260809 01:55:33.988153 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "13b011436ec644288b1ccac55b6c19f5"
format_stamp: "Formatted at 2026-08-09 01:55:33 on dist-test-slave-gkw7"
server_key: "f3e12311f65e800d7821defee66b392b"
server_key_iv: "c52711d4f0e123181a0973e63bf84097"
server_key_version: "encryptionkey@0"
I20260809 01:55:33.992364 26287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 01:55:33.994930 26308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:33.996218 26287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:55:33.996382 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "13b011436ec644288b1ccac55b6c19f5"
format_stamp: "Formatted at 2026-08-09 01:55:33 on dist-test-slave-gkw7"
server_key: "f3e12311f65e800d7821defee66b392b"
server_key_iv: "c52711d4f0e123181a0973e63bf84097"
server_key_version: "encryptionkey@0"
I20260809 01:55:33.996488 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:34.014879 26287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:34.017951 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:34.018162 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:34.025126 26287 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:35731
I20260809 01:55:34.025436 26376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:35731 every 8 connection(s)
I20260809 01:55:34.026132 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:34.028939 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:34.032569 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 26287
I20260809 01:55:34.032704 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:55:34.033010 3952 external_mini_cluster.cc:1428] Setting key d9cb093bdc74aa27520bf4d4cc411301
I20260809 01:55:34.035476 26377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Bootstrap starting.
I20260809 01:55:34.038254 26377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:34.039124 26377 log.cc:824] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:34.041432 26377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: No bootstrap required, opened a new log
Aug 09 01:55:34 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:34.045106 26377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:55:34.045361 26377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:34.045456 26377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b011436ec644288b1ccac55b6c19f5, State: Initialized, Role: FOLLOWER
I20260809 01:55:34.045917 26377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:55:34.046075 26377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:55:34.046207 26377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:55:34.046330 26377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:34.047509 26377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:55:34.047865 26377 leader_election.cc:302] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13b011436ec644288b1ccac55b6c19f5; no voters:
I20260809 01:55:34.048215 26377 leader_election.cc:288] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:55:34.048342 26384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:34.048518 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.034495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.3.220.62:35731)
0809 01:55:34.034913 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.034933 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.034977 (+ 44us) server_negotiation.cc:407] Connection header received
0809 01:55:34.035827 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.035851 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.036272 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.036646 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:34.037321 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.038834 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.039728 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.040278 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.040399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.043326 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:34.043331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:34.043345 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:34.043372 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:34.045135 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.045669 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.045680 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.045685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.045849 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.046167 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.046173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.046174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.046714 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:34.046926 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.047169 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.047495 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260809 01:55:34.048611 26384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 1 LEADER]: Becoming Leader. State: Replica: 13b011436ec644288b1ccac55b6c19f5, State: Running, Role: LEADER
I20260809 01:55:34.048947 26384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:55:34.049460 26377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:34.051333 26386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b011436ec644288b1ccac55b6c19f5. Latest consensus state: current_term: 1 leader_uuid: "13b011436ec644288b1ccac55b6c19f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } } }
I20260809 01:55:34.051456 26386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:34.051795 26388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13b011436ec644288b1ccac55b6c19f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } } }
I20260809 01:55:34.051898 26388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:34.052121 26404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:34.054935 26404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:34.060417 26404 catalog_manager.cc:1347] Generated new cluster ID: 73245ac378a6474493beac282a63d32c
I20260809 01:55:34.060484 26404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:55:34.080456 26404 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:55:34.081830 26404 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:55:34.088017 26404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Generated new TSK 0
I20260809 01:55:34.088860 26404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:55:34.121169 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35731
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:34.235471 26415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:34.235719 26415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:34.235762 26415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:34.239764 26415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:34.239822 26415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:34.239877 26415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:55:34.244982 26415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:34.246016 26415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:34.246991 26415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:34.256901 26426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:34.256909 26423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:34.256909 26424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:34.256958 26415 server_base.cc:1034] running on GCE node
I20260809 01:55:34.257627 26415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:34.258208 26415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:34.259430 26415 hybrid_clock.cc:648] HybridClock initialized: now 1786240534259410 us; error 32 us; skew 500 ppm
Aug 09 01:55:34 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:34.262331 26415 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:55:34.263229 26415 webserver.cc:460] Webserver started at http://127.3.220.1:41339/ using document root <none> and password file <none>
I20260809 01:55:34.263816 26415 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:34.263921 26415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:34.264176 26415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:34.266345 26415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "f4937de0d510453ba2922a146552986c"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "fe65dfc2a5089320f7c014acfd6bbd4a"
server_key_iv: "2088a833bd9a0f1fa1f9afb5be5cc852"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.266918 26415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "f4937de0d510453ba2922a146552986c"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "fe65dfc2a5089320f7c014acfd6bbd4a"
server_key_iv: "2088a833bd9a0f1fa1f9afb5be5cc852"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.270713 26415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 01:55:34.273197 26440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.274437 26415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:55:34.274571 26415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "f4937de0d510453ba2922a146552986c"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "fe65dfc2a5089320f7c014acfd6bbd4a"
server_key_iv: "2088a833bd9a0f1fa1f9afb5be5cc852"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.274683 26415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:34.309231 26415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:34.312701 26415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:34.312917 26415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:34.313555 26415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:34.314396 26415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:34.314467 26415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.314545 26415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:34.314589 26415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.323292 26415 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:42651
I20260809 01:55:34.323324 26604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:42651 every 8 connection(s)
I20260809 01:55:34.324178 26415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:55:34.328761 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 26415
I20260809 01:55:34.328886 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:55:34.329196 3952 external_mini_cluster.cc:1428] Setting key d44ff5e88f22b90addea3e86d7419760
Aug 09 01:55:34 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:34.338822 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.325953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:60669 (local address 127.3.220.62:35731)
0809 01:55:34.326089 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.326093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.326921 (+ 828us) server_negotiation.cc:407] Connection header received
0809 01:55:34.328250 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.328254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.328323 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.328426 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:34.329854 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.330531 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.331716 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.332132 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.332259 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.334666 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:34.334673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:34.334679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:34.334713 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:34.336378 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.337023 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.337026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.337028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.337104 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.337653 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.337656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.337657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.337838 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:34.337928 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.338587 (+ 659us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.338704 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:55:34.339985 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.326350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:60669)
0809 01:55:34.326766 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.326816 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:34.328023 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:34.328617 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:34.328642 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:34.329092 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:34.329689 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.329711 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.330667 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.330670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.331587 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.331598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.332279 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.332282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.332427 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.333033 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:34.333049 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:34.334490 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:34.336513 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:34.336523 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:34.336540 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:34.336918 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:34.337235 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:34.337241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:34.337246 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:34.337456 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:34.337961 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:34.337969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:34.338284 (+ 315us) client_negotiation.cc:769] Sending connection context
0809 01:55:34.338529 (+ 245us) client_negotiation.cc:240] Negotiation successful
0809 01:55:34.338886 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:55:34.341405 26605 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35731
I20260809 01:55:34.341671 26605 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:34.342190 26605 heartbeater.cc:511] Master 127.3.220.62:35731 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:55:34.343869 26328 ts_manager.cc:194] Registered new tserver with Master: f4937de0d510453ba2922a146552986c (127.3.220.1:42651)
I20260809 01:55:34.345510 26328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:60669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:55:34.360011 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35731
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:34.480589 26610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:34.480873 26610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:34.480952 26610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:34.485126 26610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:34.485208 26610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:34.485307 26610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:55:34.490754 26610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:34.491835 26610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:34.492856 26610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:34.502709 26618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:34.502733 26610 server_base.cc:1034] running on GCE node
W20260809 01:55:34.502698 26621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:34.502698 26616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:34.503320 26610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:34.504022 26610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:34.505218 26610 hybrid_clock.cc:648] HybridClock initialized: now 1786240534505202 us; error 27 us; skew 500 ppm
Aug 09 01:55:34 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:34.507988 26610 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:55:34.508876 26610 webserver.cc:460] Webserver started at http://127.3.220.2:39269/ using document root <none> and password file <none>
I20260809 01:55:34.509433 26610 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:34.509526 26610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:34.509789 26610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:34.511763 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "35b18d6d3e6146df8386d501a3627c06"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "659c65619a12f69d9d20d27e1a3988bc"
server_key_iv: "01017d734fb37e67e4174a1f2f034b99"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.512326 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "35b18d6d3e6146df8386d501a3627c06"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "659c65619a12f69d9d20d27e1a3988bc"
server_key_iv: "01017d734fb37e67e4174a1f2f034b99"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.516047 26610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:55:34.518513 26631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.519665 26610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:55:34.519799 26610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "35b18d6d3e6146df8386d501a3627c06"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "659c65619a12f69d9d20d27e1a3988bc"
server_key_iv: "01017d734fb37e67e4174a1f2f034b99"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.519912 26610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:34.535962 26610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:34.538959 26610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:34.539175 26610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:34.539760 26610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:34.540639 26610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:34.540710 26610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.540783 26610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:34.540827 26610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.550153 26610 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:34267
I20260809 01:55:34.550172 26787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:34267 every 8 connection(s)
I20260809 01:55:34.551151 26610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:55:34.556735 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 26610
I20260809 01:55:34.556878 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:55:34.557173 3952 external_mini_cluster.cc:1428] Setting key 4fb64f4bb038dcb7b70af8543013a296
Aug 09 01:55:34 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:34.565409 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.554518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:42243 (local address 127.3.220.62:35731)
0809 01:55:34.554691 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.554696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.554719 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:55:34.555682 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.555686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.555745 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.555825 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:34.557064 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.557757 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.558935 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.559182 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.559296 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.561848 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:34.561851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:34.561853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:34.561878 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:34.563334 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.563870 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.563873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.563874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.563951 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.564305 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.564310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.564311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.564497 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:34.564611 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.565098 (+ 487us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.565238 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 01:55:34.566208 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.553630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:42243)
0809 01:55:34.554145 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.554187 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:34.555450 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:34.556001 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:34.556008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:34.556365 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:34.556901 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.556922 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.557889 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.557892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.558799 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.558808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.559336 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.559340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.559523 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.560144 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:34.560160 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:34.561663 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:34.563463 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:34.563468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:34.563483 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:34.563746 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:34.564046 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:34.564050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:34.564052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:34.564205 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:34.564616 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:34.564625 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:34.564864 (+ 239us) client_negotiation.cc:769] Sending connection context
0809 01:55:34.565053 (+ 189us) client_negotiation.cc:240] Negotiation successful
0809 01:55:34.565368 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":151,"threads_started":1}
I20260809 01:55:34.567549 26788 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35731
I20260809 01:55:34.567759 26788 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:55:34.568213 26788 heartbeater.cc:511] Master 127.3.220.62:35731 requested a full tablet report, sending...
I20260809 01:55:34.569283 26328 ts_manager.cc:194] Registered new tserver with Master: 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
I20260809 01:55:34.569929 26328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:42243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:55:34.588845 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35731
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:34.703346 26796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:34.703599 26796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:34.703641 26796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:34.708319 26796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:34.708380 26796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:34.708493 26796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:55:34.713918 26796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:34.715054 26796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:34.716082 26796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:34.725903 26813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:34.725917 26815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:34.725870 26810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:34.726132 26796 server_base.cc:1034] running on GCE node
I20260809 01:55:34.726593 26796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:34.727233 26796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:34.728389 26796 hybrid_clock.cc:648] HybridClock initialized: now 1786240534728362 us; error 48 us; skew 500 ppm
Aug 09 01:55:34 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:34.731281 26796 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:55:34.732076 26796 webserver.cc:460] Webserver started at http://127.3.220.3:37029/ using document root <none> and password file <none>
I20260809 01:55:34.732631 26796 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:34.732694 26796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:34.732861 26796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:55:34.734881 26796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "711c58ba94c24ee4a0391f7409d89a8f"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "794e06c06c51f6f7c1294a98be8730f4"
server_key_iv: "24769ca62c531bbdabe7713c9d166447"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.735466 26796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "711c58ba94c24ee4a0391f7409d89a8f"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "794e06c06c51f6f7c1294a98be8730f4"
server_key_iv: "24769ca62c531bbdabe7713c9d166447"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.739099 26796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:55:34.741460 26826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.742765 26796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:55:34.742918 26796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "711c58ba94c24ee4a0391f7409d89a8f"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "794e06c06c51f6f7c1294a98be8730f4"
server_key_iv: "24769ca62c531bbdabe7713c9d166447"
server_key_version: "encryptionkey@0"
I20260809 01:55:34.743026 26796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:34.757316 26796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:34.760201 26796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:34.760417 26796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:34.761034 26796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:34.761986 26796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:34.762035 26796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.762106 26796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:34.762152 26796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:34.770970 26796 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:44243
I20260809 01:55:34.771065 26983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:44243 every 8 connection(s)
I20260809 01:55:34.771872 26796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:55:34.776021 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 26796
I20260809 01:55:34.776170 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:55:34.776500 3952 external_mini_cluster.cc:1428] Setting key 53642cea467bdcddeb0360b294ad1ade
Aug 09 01:55:34 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:34.785983 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.773721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:55397 (local address 127.3.220.62:35731)
0809 01:55:34.773879 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.773883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.774795 (+ 912us) server_negotiation.cc:407] Connection header received
0809 01:55:34.776138 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.776143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.776204 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.776313 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:34.777714 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.778428 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.779721 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.779939 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.780016 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.782491 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:34.782495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:34.782497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:34.782519 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:34.783903 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.784405 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.784409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.784410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.784483 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.784839 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.784843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.784844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.785054 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:34.785163 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.785698 (+ 535us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.785818 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 01:55:34.786717 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.774151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:55397)
0809 01:55:34.774627 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.774690 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:34.775925 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:34.776447 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:34.776454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:34.776854 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:34.777509 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.777570 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.778560 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.778567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.779606 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.779616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.780065 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.780068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.780229 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.780739 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:34.780754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:34.782331 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:34.784022 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:34.784028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:34.784041 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:34.784291 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:34.784575 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:34.784579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:34.784581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:34.784715 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:34.785162 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:34.785168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:34.785435 (+ 267us) client_negotiation.cc:769] Sending connection context
0809 01:55:34.785627 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 01:55:34.785961 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":149,"threads_started":1}
I20260809 01:55:34.787732 26984 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35731
I20260809 01:55:34.787962 26984 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:34.788367 26984 heartbeater.cc:511] Master 127.3.220.62:35731 requested a full tablet report, sending...
I20260809 01:55:34.789357 26328 ts_manager.cc:194] Registered new tserver with Master: 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243)
I20260809 01:55:34.789983 26328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:55397
I20260809 01:55:34.790521 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:55:34.799312 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.792547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.3.220.62:35731)
0809 01:55:34.792681 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.792686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.792903 (+ 217us) server_negotiation.cc:407] Connection header received
0809 01:55:34.792984 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.792986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.793020 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.793078 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:34.793767 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.794227 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.795021 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.795214 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.795286 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.796444 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:34.796447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:34.796449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:34.796469 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:34.797876 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.798227 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.798229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.798230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.798309 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:34.798620 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:34.798622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:34.798623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:34.798806 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:34.798902 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.799119 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.799202 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 01:55:34.802309 26327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:55:34.805207 26327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:55:34.820106 26999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.813709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:44243 (local address 127.0.0.1:46486)
0809 01:55:34.814018 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.814063 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:34.814178 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:34.815036 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:34.815043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:34.815071 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:34.815260 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.815271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.816936 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.816939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.818528 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.818541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.819574 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.819577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.819713 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.819782 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 01:55:34.819854 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 01:55:34.819962 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":239,"thread_start_us":85,"threads_started":1}
I20260809 01:55:34.820127 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.814992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.3.220.1:42651)
0809 01:55:34.815575 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.815604 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.815628 (+ 24us) server_negotiation.cc:407] Connection header received
0809 01:55:34.815710 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.815715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.815896 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.816062 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:34.816272 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.817249 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.818784 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.819343 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.819485 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.819636 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.819840 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.819949 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":106,"threads_started":1}
I20260809 01:55:34.820156 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.814571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.3.220.2:34267)
0809 01:55:34.815001 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.815012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.815131 (+ 119us) server_negotiation.cc:407] Connection header received
0809 01:55:34.815385 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.815390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.815533 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.815687 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:34.815887 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.816725 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.818445 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.819209 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.819333 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.819476 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.819918 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.820011 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":108,"threads_started":1}
I20260809 01:55:34.820519 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.813808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.3.220.3:44243)
0809 01:55:34.814464 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.814480 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.814507 (+ 27us) server_negotiation.cc:407] Connection header received
0809 01:55:34.814590 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.814596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.814824 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.815445 (+ 621us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:34.815611 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.816764 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.818681 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.819450 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.819924 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.820054 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.820098 (+ 44us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.820370 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":176,"threads_started":1}
I20260809 01:55:34.820675 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.814741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:42651 (local address 127.0.0.1:35306)
0809 01:55:34.815129 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.815141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:34.815217 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:34.816051 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:34.816054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:34.816074 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:34.816159 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.816164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.817392 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.817396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.818640 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.818651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.819484 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.819488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.819612 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.819658 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 01:55:34.819808 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 01:55:34.820483 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"spinlock_wait_cycles":4224,"thread_start_us":103,"threads_started":1}
I20260809 01:55:34.820822 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.814412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:34267 (local address 127.0.0.1:55792)
0809 01:55:34.815037 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.815051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:34.815214 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:34.815667 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:34.815669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:34.815687 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:34.815782 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.815789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.816866 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.816869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.818327 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.818336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.819546 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.819553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.819708 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.819783 (+ 75us) client_negotiation.cc:769] Sending connection context
0809 01:55:34.820554 (+ 771us) client_negotiation.cc:240] Negotiation successful
0809 01:55:34.820645 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":210,"spinlock_wait_cycles":384,"thread_start_us":95,"threads_started":1}
I20260809 01:55:34.823539 26507 tablet_service.cc:1467] Processing CreateTablet for tablet 92df992e224547f098b870697d75a55b (DEFAULT_TABLE table=test-table [id=1803c9be074c45148846009fd3261ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:34.823529 26699 tablet_service.cc:1467] Processing CreateTablet for tablet 92df992e224547f098b870697d75a55b (DEFAULT_TABLE table=test-table [id=1803c9be074c45148846009fd3261ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:34.823529 26892 tablet_service.cc:1467] Processing CreateTablet for tablet 92df992e224547f098b870697d75a55b (DEFAULT_TABLE table=test-table [id=1803c9be074c45148846009fd3261ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:34.824585 26699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92df992e224547f098b870697d75a55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:34.824589 26892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92df992e224547f098b870697d75a55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:34.824594 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92df992e224547f098b870697d75a55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:34.831141 27008 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Bootstrap starting.
I20260809 01:55:34.832122 27010 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: Bootstrap starting.
I20260809 01:55:34.833827 27009 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: Bootstrap starting.
I20260809 01:55:34.833946 27008 tablet_bootstrap.cc:654] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:34.834434 27010 tablet_bootstrap.cc:654] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:34.835428 27008 log.cc:824] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:34.835731 27010 log.cc:824] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:34.835994 27009 tablet_bootstrap.cc:654] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:34.836797 27009 log.cc:824] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:34.838083 27010 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: No bootstrap required, opened a new log
I20260809 01:55:34.838328 27010 ts_tablet_manager.cc:1397] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:55:34.838368 27008 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: No bootstrap required, opened a new log
I20260809 01:55:34.838549 27008 ts_tablet_manager.cc:1397] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260809 01:55:34.838755 27009 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: No bootstrap required, opened a new log
I20260809 01:55:34.838936 27009 ts_tablet_manager.cc:1397] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:55:34.841645 27010 raft_consensus.cc:348] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.841928 27010 raft_consensus.cc:374] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:34.842001 27010 raft_consensus.cc:729] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 711c58ba94c24ee4a0391f7409d89a8f, State: Initialized, Role: FOLLOWER
I20260809 01:55:34.842465 27010 consensus_queue.cc:260] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.842427 27008 raft_consensus.cc:348] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.842622 27009 raft_consensus.cc:348] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.842865 27009 raft_consensus.cc:374] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:34.842865 27008 raft_consensus.cc:374] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:34.842921 27008 raft_consensus.cc:729] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4937de0d510453ba2922a146552986c, State: Initialized, Role: FOLLOWER
I20260809 01:55:34.842931 27009 raft_consensus.cc:729] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b18d6d3e6146df8386d501a3627c06, State: Initialized, Role: FOLLOWER
I20260809 01:55:34.843384 27009 consensus_queue.cc:260] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.843384 27008 consensus_queue.cc:260] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.843549 27010 ts_tablet_manager.cc:1428] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:55:34.843926 26984 heartbeater.cc:503] Master 127.3.220.62:35731 was elected leader, sending a full tablet report...
I20260809 01:55:34.844033 26788 heartbeater.cc:503] Master 127.3.220.62:35731 was elected leader, sending a full tablet report...
I20260809 01:55:34.844276 27009 ts_tablet_manager.cc:1428] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260809 01:55:34.844554 26605 heartbeater.cc:503] Master 127.3.220.62:35731 was elected leader, sending a full tablet report...
I20260809 01:55:34.845044 27008 ts_tablet_manager.cc:1428] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:55:34.859860 27015 raft_consensus.cc:482] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:34.860085 27015 raft_consensus.cc:504] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.861210 27015 leader_election.cc:288] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267), f4937de0d510453ba2922a146552986c (127.3.220.1:42651)
I20260809 01:55:34.865056 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.861394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:34267 (local address 127.3.220.3:33031)
0809 01:55:34.861552 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.861593 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:34.861682 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:34.861967 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:34.861970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:34.861990 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:34.862125 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.862132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.863141 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.863146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.863912 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.863921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.864631 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.864635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.864744 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.864761 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 01:55:34.864852 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 01:55:34.864924 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 01:55:34.865056 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.861537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:33031 (local address 127.3.220.2:34267)
0809 01:55:34.861693 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.861698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.861714 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:55:34.861771 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.861774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.861828 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.862248 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:34.862376 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.863033 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.864090 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.864504 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.864630 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.864690 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.864866 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.864938 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 01:55:34.865294 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.861880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:45123 (local address 127.3.220.1:42651)
0809 01:55:34.862009 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.862014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.862217 (+ 203us) server_negotiation.cc:407] Connection header received
0809 01:55:34.862388 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.862390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.862432 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.862494 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:34.862762 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.863264 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.864348 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.864750 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.864870 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.864927 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.865126 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.865193 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:55:34.865304 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.861720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:42651 (local address 127.3.220.3:45123)
0809 01:55:34.862137 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.862149 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:34.862278 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:34.862531 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:34.862534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:34.862552 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:34.862658 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.862664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.863393 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.863396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.864226 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:34.864235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.864881 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.864885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:34.864987 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.865011 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 01:55:34.865100 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 01:55:34.865168 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":152,"threads_started":1}
I20260809 01:55:34.865967 26726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92df992e224547f098b870697d75a55b" candidate_uuid: "711c58ba94c24ee4a0391f7409d89a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b18d6d3e6146df8386d501a3627c06" is_pre_election: true
I20260809 01:55:34.865969 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92df992e224547f098b870697d75a55b" candidate_uuid: "711c58ba94c24ee4a0391f7409d89a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4937de0d510453ba2922a146552986c" is_pre_election: true
I20260809 01:55:34.866255 26538 raft_consensus.cc:2454] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 711c58ba94c24ee4a0391f7409d89a8f in term 0.
I20260809 01:55:34.866255 26726 raft_consensus.cc:2454] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 711c58ba94c24ee4a0391f7409d89a8f in term 0.
I20260809 01:55:34.866706 26828 leader_election.cc:302] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 711c58ba94c24ee4a0391f7409d89a8f, f4937de0d510453ba2922a146552986c; no voters:
I20260809 01:55:34.866931 27015 raft_consensus.cc:2781] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:55:34.867038 27015 raft_consensus.cc:482] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:55:34.867110 27015 raft_consensus.cc:3037] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:34.868161 27015 raft_consensus.cc:504] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.868489 27015 leader_election.cc:288] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [CANDIDATE]: Term 1 election: Requested vote from peers 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267), f4937de0d510453ba2922a146552986c (127.3.220.1:42651)
I20260809 01:55:34.868854 26726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92df992e224547f098b870697d75a55b" candidate_uuid: "711c58ba94c24ee4a0391f7409d89a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b18d6d3e6146df8386d501a3627c06"
I20260809 01:55:34.868984 26726 raft_consensus.cc:3037] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:34.869024 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92df992e224547f098b870697d75a55b" candidate_uuid: "711c58ba94c24ee4a0391f7409d89a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4937de0d510453ba2922a146552986c"
I20260809 01:55:34.869205 26538 raft_consensus.cc:3037] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:34.870162 26726 raft_consensus.cc:2454] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 711c58ba94c24ee4a0391f7409d89a8f in term 1.
I20260809 01:55:34.870402 26538 raft_consensus.cc:2454] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 711c58ba94c24ee4a0391f7409d89a8f in term 1.
I20260809 01:55:34.870451 26828 leader_election.cc:302] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b18d6d3e6146df8386d501a3627c06, 711c58ba94c24ee4a0391f7409d89a8f; no voters:
I20260809 01:55:34.870766 27015 raft_consensus.cc:2781] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:34.871035 27015 raft_consensus.cc:686] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 1 LEADER]: Becoming Leader. State: Replica: 711c58ba94c24ee4a0391f7409d89a8f, State: Running, Role: LEADER
I20260809 01:55:34.871361 27015 consensus_queue.cc:237] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:34.874732 26328 catalog_manager.cc:5495] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3). New cstate: current_term: 1 leader_uuid: "711c58ba94c24ee4a0391f7409d89a8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } health_report { overall_health: UNKNOWN } } }
I20260809 01:55:34.898772 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.895342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46490 (local address 127.3.220.3:44243)
0809 01:55:34.895476 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:34.895480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:34.895493 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:55:34.895602 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:34.895605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:34.895660 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:34.895748 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:55:34.895967 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.896462 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.897338 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:34.897606 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:34.897684 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:34.897957 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:55:34.898358 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:55:34.898447 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:34.898521 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 01:55:34.898567 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 01:55:34.904819 26726 raft_consensus.cc:1261] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 1 FOLLOWER]: Refusing update from remote peer 711c58ba94c24ee4a0391f7409d89a8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:34.904816 26538 raft_consensus.cc:1261] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Refusing update from remote peer 711c58ba94c24ee4a0391f7409d89a8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:34.905551 27015 consensus_queue.cc:1035] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:34.905730 27019 consensus_queue.cc:1035] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:34.913884 27027 mvcc.cc:205] Tried to move back new op lower bound from 7316441230963040256 to 7316441230836019200. Current Snapshot: MvccSnapshot[applied={T|T < 7316441230963040256}]
I20260809 01:55:34.914645 27026 mvcc.cc:205] Tried to move back new op lower bound from 7316441230963040256 to 7316441230836019200. Current Snapshot: MvccSnapshot[applied={T|T < 7316441230963040256}]
I20260809 01:55:34.921578 26328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:34.921726 26328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:34.923964 26328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Sending DeleteTablet for 3 replicas of tablet 92df992e224547f098b870697d75a55b
I20260809 01:55:34.924860 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 26415
I20260809 01:55:34.924839 26892 tablet_service.cc:1514] Processing DeleteTablet for tablet 92df992e224547f098b870697d75a55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:34 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:46486
I20260809 01:55:34.925014 26507 tablet_service.cc:1514] Processing DeleteTablet for tablet 92df992e224547f098b870697d75a55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:34 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:35306
I20260809 01:55:34.928714 26699 tablet_service.cc:1514] Processing DeleteTablet for tablet 92df992e224547f098b870697d75a55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:34 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:55792
I20260809 01:55:34.930173 27043 tablet_replica.cc:330] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: stopping tablet replica
I20260809 01:55:34.930508 27043 raft_consensus.cc:2229] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:34.930842 27043 raft_consensus.cc:2258] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:34.932480 27042 tablet_replica.cc:330] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: stopping tablet replica
I20260809 01:55:34.932781 27042 raft_consensus.cc:2229] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:55:34.933194 27042 raft_consensus.cc:2258] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:34.933416 27043 ts_tablet_manager.cc:1905] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:34.934880 27042 ts_tablet_manager.cc:1905] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:34.936372 27043 ts_tablet_manager.cc:1918] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:34.936537 27043 log.cc:1196] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/92df992e224547f098b870697d75a55b
I20260809 01:55:34.936965 27043 ts_tablet_manager.cc:1939] T 92df992e224547f098b870697d75a55b P 35b18d6d3e6146df8386d501a3627c06: Deleting consensus metadata
I20260809 01:55:34.938333 26311 catalog_manager.cc:4841] TS 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267): tablet 92df992e224547f098b870697d75a55b (table test-table [id=1803c9be074c45148846009fd3261ec2]) successfully deleted
I20260809 01:55:34.938460 27042 ts_tablet_manager.cc:1918] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:34.938560 27042 log.cc:1196] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/92df992e224547f098b870697d75a55b
I20260809 01:55:34.938918 27042 ts_tablet_manager.cc:1939] T 92df992e224547f098b870697d75a55b P 711c58ba94c24ee4a0391f7409d89a8f: Deleting consensus metadata
I20260809 01:55:34.940081 26311 catalog_manager.cc:4841] TS 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243): tablet 92df992e224547f098b870697d75a55b (table test-table [id=1803c9be074c45148846009fd3261ec2]) successfully deleted
W20260809 01:55:34.942735 26311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:42651 (error 108)
I20260809 01:55:34.942777 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 26610
I20260809 01:55:34.947355 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:34.947030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:42651 (local address 127.0.0.1:35322)
0809 01:55:34.947188 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:34.947264 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 01:55:34.947531 26311 catalog_manager.cc:4584] TS f4937de0d510453ba2922a146552986c (127.3.220.1:42651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92df992e224547f098b870697d75a55b: Network error: Client connection negotiation failed: client connection to 127.3.220.1:42651: connect: Connection refused (error 111)
I20260809 01:55:34.952594 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 26796
I20260809 01:55:34.964367 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 26287
I20260809 01:55:54.977231 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35731
--webserver_interface=127.3.220.62
--webserver_port=35477
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:55:55.091212 27045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:55.091512 27045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:55.091609 27045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:55.095597 27045 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 01:55:55.095681 27045 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 01:55:55.095711 27045 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 01:55:55.095746 27045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:55:55.095786 27045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:55.095846 27045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:55.095886 27045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:55:55.095923 27045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:55:55.101016 27045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:55.102087 27045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:55.103093 27045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:55:55.117694 27045 server_base.cc:1034] running on GCE node
W20260809 01:55:55.117677 27051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:55.117628 27058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:55.117621 27054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:55.118623 27045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:55.119863 27045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:55.121065 27045 hybrid_clock.cc:648] HybridClock initialized: now 1786240555121025 us; error 51 us; skew 500 ppm
Aug 09 01:55:55 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:55.124046 27045 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:55:55.124986 27045 webserver.cc:460] Webserver started at http://127.3.220.62:35477/ using document root <none> and password file <none>
I20260809 01:55:55.125655 27045 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:55.125764 27045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:55.129850 27045 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:55:55.132323 27068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:55.133786 27045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:55:55.133947 27045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "13b011436ec644288b1ccac55b6c19f5"
format_stamp: "Formatted at 2026-08-09 01:55:33 on dist-test-slave-gkw7"
server_key: "f3e12311f65e800d7821defee66b392b"
server_key_iv: "c52711d4f0e123181a0973e63bf84097"
server_key_version: "encryptionkey@0"
I20260809 01:55:55.134614 27045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:55.160199 27045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:55.163661 27045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:55.163892 27045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:55.171716 27045 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:35731
I20260809 01:55:55.172001 27143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:35731 every 8 connection(s)
I20260809 01:55:55.172945 27045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:55:55.175508 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 27045
I20260809 01:55:55.175973 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:42651
--local_ip_for_outbound_sockets=127.3.220.1
--tserver_master_addrs=127.3.220.62:35731
--webserver_port=41339
--webserver_interface=127.3.220.1
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:55.177687 27144 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:55:55.180161 27144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Bootstrap starting.
I20260809 01:55:55.205034 27144 log.cc:824] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:55.216675 27144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:55:55.217172 27144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Bootstrap complete.
I20260809 01:55:55.223142 27144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:55:55.223673 27144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b011436ec644288b1ccac55b6c19f5, State: Initialized, Role: FOLLOWER
I20260809 01:55:55.224251 27144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:55:55.224407 27144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:55:55.224481 27144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:55:55.224579 27144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 1 FOLLOWER]: Advancing to term 2
I20260809 01:55:55.227257 27144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:55:55.227640 27144 leader_election.cc:302] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13b011436ec644288b1ccac55b6c19f5; no voters:
I20260809 01:55:55.228013 27144 leader_election.cc:288] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 01:55:55.228143 27148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 01:55:55.228463 27148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 2 LEADER]: Becoming Leader. State: Replica: 13b011436ec644288b1ccac55b6c19f5, State: Running, Role: LEADER
I20260809 01:55:55.228825 27148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:55:55.229377 27144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:55:55.230280 27149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "13b011436ec644288b1ccac55b6c19f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } } }
I20260809 01:55:55.230405 27149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:55.230772 27150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b011436ec644288b1ccac55b6c19f5. Latest consensus state: current_term: 2 leader_uuid: "13b011436ec644288b1ccac55b6c19f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } } }
I20260809 01:55:55.230937 27150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:55:55.231264 27166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:55:55.236333 27166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:55:55.236979 27166 catalog_manager.cc:1259] Loaded cluster ID: 73245ac378a6474493beac282a63d32c
I20260809 01:55:55.237044 27166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:55:55.239746 27166 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:55:55.240518 27166 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Loaded TSK: 0
I20260809 01:55:55.259817 27166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Generated new TSK 1
I20260809 01:55:55.260094 27166 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 01:55:55.318202 27146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:55.318488 27146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:55.318581 27146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:55.322588 27146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:55.322672 27146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:55.322762 27146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:55:55.327993 27146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:42651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=41339
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:55.329052 27146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:55.330281 27146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:55.341405 27188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:55.341454 27192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:55.341720 27146 server_base.cc:1034] running on GCE node
W20260809 01:55:55.341454 27189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:55.342319 27146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:55.342971 27146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:55.344156 27146 hybrid_clock.cc:648] HybridClock initialized: now 1786240555344126 us; error 53 us; skew 500 ppm
Aug 09 01:55:55 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:55.346982 27146 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:55:55.347945 27146 webserver.cc:460] Webserver started at http://127.3.220.1:41339/ using document root <none> and password file <none>
I20260809 01:55:55.348510 27146 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:55.348585 27146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:55.352600 27146 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:55:55.354887 27202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:55.356200 27146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:55:55.356340 27146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "f4937de0d510453ba2922a146552986c"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "fe65dfc2a5089320f7c014acfd6bbd4a"
server_key_iv: "2088a833bd9a0f1fa1f9afb5be5cc852"
server_key_version: "encryptionkey@0"
I20260809 01:55:55.356878 27146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:55.376905 27146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:55.380314 27146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:55.380532 27146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:55.381204 27146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:55.382455 27212 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 01:55:55.386616 27146 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 01:55:55.386682 27146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260809 01:55:55.386754 27146 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 01:55:55.388407 27146 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 01:55:55.388468 27146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 01:55:55.388725 27212 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Bootstrap starting.
I20260809 01:55:55.399917 27146 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:42651
I20260809 01:55:55.399940 27366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:42651 every 8 connection(s)
I20260809 01:55:55.401224 27146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:55:55.407716 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 27146
I20260809 01:55:55.408211 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:34267
--local_ip_for_outbound_sockets=127.3.220.2
--tserver_master_addrs=127.3.220.62:35731
--webserver_port=39269
--webserver_interface=127.3.220.2
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:55:55.410326 27212 log.cc:824] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:55:55 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:55.418783 27212 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:55:55.419229 27212 tablet_bootstrap.cc:492] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Bootstrap complete.
I20260809 01:55:55.420651 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.403526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:47523)
0809 01:55:55.404037 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:55.404079 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:55.404904 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:55.406080 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:55.406088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:55.406526 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:55.406986 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.406999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.408758 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.408762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.410066 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.410078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.411174 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.411178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.411352 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.412037 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:55.412062 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:55.414046 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:55.416905 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:55.416911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:55.416922 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:55.417269 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:55.417759 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:55.417762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:55.417764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:55.417930 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:55.418733 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:55.418738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:55.419049 (+ 311us) client_negotiation.cc:769] Sending connection context
0809 01:55:55.419284 (+ 235us) client_negotiation.cc:240] Negotiation successful
0809 01:55:55.419623 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260809 01:55:55.420704 27212 ts_tablet_manager.cc:1397] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.008s
I20260809 01:55:55.421147 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.403477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:47523 (local address 127.3.220.62:35731)
0809 01:55:55.403859 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.403880 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.404209 (+ 329us) server_negotiation.cc:407] Connection header received
0809 01:55:55.405326 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.405337 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.405699 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.406027 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:55.407172 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.408580 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.410234 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.411004 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.411207 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.414281 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:55.414291 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:55.414314 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:55.414357 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:55.416755 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.417406 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.417416 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.417423 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.417630 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.418045 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.418051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.418055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.418587 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:55.418752 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.419476 (+ 724us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.419845 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":128,"threads_started":1}
I20260809 01:55:55.423890 27369 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35731
I20260809 01:55:55.423547 27212 raft_consensus.cc:348] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.424032 27212 raft_consensus.cc:729] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4937de0d510453ba2922a146552986c, State: Initialized, Role: FOLLOWER
I20260809 01:55:55.424129 27369 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:55.424500 27212 consensus_queue.cc:260] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.424660 27369 heartbeater.cc:511] Master 127.3.220.62:35731 requested a full tablet report, sending...
I20260809 01:55:55.425518 27212 ts_tablet_manager.cc:1428] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:55:55.427392 27093 ts_manager.cc:194] Registered new tserver with Master: f4937de0d510453ba2922a146552986c (127.3.220.1:42651)
I20260809 01:55:55.431581 27093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:47523
I20260809 01:55:55.434196 27369 heartbeater.cc:503] Master 127.3.220.62:35731 was elected leader, sending a full tablet report...
Aug 09 01:55:55 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.1@KRBTEST.COM
I20260809 01:55:55.442452 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.430916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:42651 (local address 127.0.0.1:33356)
0809 01:55:55.431473 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:55.431517 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:55.431646 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:55.432313 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:55.432319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:55.432732 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:55.432866 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.432875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.434418 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.434421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.435402 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.435415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.436117 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.436121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.436238 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.436841 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:55.436856 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:55.438429 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:55.440460 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:55.440472 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:55.440477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:55.440726 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:55.441048 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:55.441051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:55.441053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:55.441153 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:55.441758 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:55.441764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:55.441904 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 01:55:55.442049 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 01:55:55.442262 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":127,"threads_started":1}
I20260809 01:55:55.442459 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.430956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.3.220.1:42651)
0809 01:55:55.431924 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.431933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.431951 (+ 18us) server_negotiation.cc:407] Connection header received
0809 01:55:55.432005 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.432009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.432152 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.432306 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:55.433005 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.434263 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.435639 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.435997 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.436084 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.438598 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:55.438607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:55.438617 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:55.438638 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:55.440336 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.440863 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.440868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.440869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.440948 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.441277 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.441280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.441282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.441631 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:55.441786 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.442148 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.442321 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":113,"threads_started":1}
I20260809 01:55:55.444039 27273 tablet_service.cc:1514] Processing DeleteTablet for tablet 92df992e224547f098b870697d75a55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:34 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:33356
I20260809 01:55:55.444996 27379 tablet_replica.cc:330] stopping tablet replica
I20260809 01:55:55.445326 27379 raft_consensus.cc:2229] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:55.445619 27379 raft_consensus.cc:2258] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:55.446751 27379 ts_tablet_manager.cc:1905] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:55.451557 27379 ts_tablet_manager.cc:1918] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:55.451663 27379 log.cc:1196] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal/wals/92df992e224547f098b870697d75a55b
I20260809 01:55:55.451977 27379 ts_tablet_manager.cc:1939] T 92df992e224547f098b870697d75a55b P f4937de0d510453ba2922a146552986c: Deleting consensus metadata
I20260809 01:55:55.453960 27071 catalog_manager.cc:4841] TS f4937de0d510453ba2922a146552986c (127.3.220.1:42651): tablet 92df992e224547f098b870697d75a55b (table test-table [id=1803c9be074c45148846009fd3261ec2]) successfully deleted
W20260809 01:55:55.534647 27373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:55.534905 27373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:55.534950 27373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:55.539354 27373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:55.539415 27373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:55.539526 27373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:55:55.544798 27373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:34267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=39269
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:55.545836 27373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:55.546916 27373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:55.557214 27387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:55.557281 27386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:55.557358 27373 server_base.cc:1034] running on GCE node
W20260809 01:55:55.557265 27389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:55.557914 27373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:55.558637 27373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:55.559823 27373 hybrid_clock.cc:648] HybridClock initialized: now 1786240555559806 us; error 26 us; skew 500 ppm
Aug 09 01:55:55 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:55.562848 27373 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:55:55.563827 27373 webserver.cc:460] Webserver started at http://127.3.220.2:39269/ using document root <none> and password file <none>
I20260809 01:55:55.564414 27373 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:55.564538 27373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:55.568538 27373 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:55:55.570748 27402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:55.572103 27373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:55:55.572311 27373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "35b18d6d3e6146df8386d501a3627c06"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "659c65619a12f69d9d20d27e1a3988bc"
server_key_iv: "01017d734fb37e67e4174a1f2f034b99"
server_key_version: "encryptionkey@0"
I20260809 01:55:55.572906 27373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:55.590008 27373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:55.593313 27373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:55.593560 27373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:55.594195 27373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:55.595054 27373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:55.595141 27373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:55.595223 27373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:55.595286 27373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:55.605579 27373 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:34267
I20260809 01:55:55.605608 27556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:34267 every 8 connection(s)
I20260809 01:55:55.606635 27373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:55:55.606961 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 27373
I20260809 01:55:55.607494 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:44243
--local_ip_for_outbound_sockets=127.3.220.3
--tserver_master_addrs=127.3.220.62:35731
--webserver_port=37029
--webserver_interface=127.3.220.3
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 01:55:55 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:55.622155 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.608699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:44313 (local address 127.3.220.62:35731)
0809 01:55:55.608884 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.608891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.609781 (+ 890us) server_negotiation.cc:407] Connection header received
0809 01:55:55.611266 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.611269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.611365 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.611503 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:55.612915 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.613515 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.614885 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.615173 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.615296 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.618235 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:55.618240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:55.618243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:55.618280 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:55.620014 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.620536 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.620540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.620542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.620633 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.620960 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.620963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.620965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.621178 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:55.621285 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.621831 (+ 546us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.621982 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 01:55:55.623011 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.609059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:44313)
0809 01:55:55.609595 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:55.609662 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:55.611019 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:55.611716 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:55.611724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:55.612160 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:55.612752 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.612767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.613711 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.613715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.614740 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.614751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.615340 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.615345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.615530 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.616188 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:55.616204 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:55.618030 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:55.620156 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:55.620160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:55.620173 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:55.620437 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:55.620728 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:55.620730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:55.620731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:55.620868 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:55.621291 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:55.621296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:55.621526 (+ 230us) client_negotiation.cc:769] Sending connection context
0809 01:55:55.621765 (+ 239us) client_negotiation.cc:240] Negotiation successful
0809 01:55:55.622041 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":158,"threads_started":1}
I20260809 01:55:55.624346 27557 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35731
I20260809 01:55:55.624593 27557 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:55.625114 27557 heartbeater.cc:511] Master 127.3.220.62:35731 requested a full tablet report, sending...
I20260809 01:55:55.626277 27093 ts_manager.cc:194] Registered new tserver with Master: 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
I20260809 01:55:55.626904 27093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:44313
W20260809 01:55:55.724869 27561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:55:55.725136 27561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:55:55.725178 27561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:55:55.729187 27561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:55:55.729250 27561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:55:55.729307 27561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:55:55.734987 27561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:44243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=37029
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:55:55.736017 27561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:55:55.737042 27561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:55:55.747010 27572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:55:55.747020 27568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:55.747179 27561 server_base.cc:1034] running on GCE node
W20260809 01:55:55.747010 27567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:55:55.747787 27561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:55:55.748481 27561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:55:55.749601 27561 hybrid_clock.cc:648] HybridClock initialized: now 1786240555749569 us; error 45 us; skew 500 ppm
Aug 09 01:55:55 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:55:55.752269 27561 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:55:55.753204 27561 webserver.cc:460] Webserver started at http://127.3.220.3:37029/ using document root <none> and password file <none>
I20260809 01:55:55.753856 27561 fs_manager.cc:359] Metadata directory not provided
I20260809 01:55:55.753973 27561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:55:55.757789 27561 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260809 01:55:55.759881 27580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:55.761291 27561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:55:55.761420 27561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "711c58ba94c24ee4a0391f7409d89a8f"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "794e06c06c51f6f7c1294a98be8730f4"
server_key_iv: "24769ca62c531bbdabe7713c9d166447"
server_key_version: "encryptionkey@0"
I20260809 01:55:55.762017 27561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:55:55.772540 27561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:55:55.775281 27561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:55:55.775472 27561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:55:55.776041 27561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:55:55.776911 27561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:55:55.776984 27561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:55.777050 27561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:55:55.777097 27561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:55:55.786196 27561 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:44243
I20260809 01:55:55.786334 27745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:44243 every 8 connection(s)
I20260809 01:55:55.787498 27561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:55:55.795341 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 27561
Aug 09 01:55:55 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:55:55.801466 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.789307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:39679 (local address 127.3.220.62:35731)
0809 01:55:55.789461 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.789465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.790403 (+ 938us) server_negotiation.cc:407] Connection header received
0809 01:55:55.791768 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.791771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.791820 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.791896 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:55.793111 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.793694 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.794777 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.794988 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.795102 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.798023 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:55.798027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:55.798030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:55.798057 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:55.799495 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.799999 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.800003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.800005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.800083 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.800423 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.800428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.800430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.800594 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:55.800712 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.801252 (+ 540us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.801345 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":1280}
I20260809 01:55:55.802438 27751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.789760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:39679)
0809 01:55:55.790244 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:55.790285 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:55.791547 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:55.792064 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:55.792071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:55.792497 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:55:55.792979 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.792992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.793813 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.793816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.794670 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.794678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.795131 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.795133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.795264 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.795892 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:55:55.795934 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:55:55.797866 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:55:55.799623 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:55.799628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:55.799642 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:55.799895 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:55.800186 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:55:55.800189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:55:55.800191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:55:55.800326 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:55:55.800710 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:55:55.800717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:55:55.800987 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 01:55:55.801208 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 01:55:55.801462 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":120,"threads_started":1}
I20260809 01:55:55.803578 27746 heartbeater.cc:347] Connected to a master server at 127.3.220.62:35731
I20260809 01:55:55.803822 27746 heartbeater.cc:464] Registering TS with master...
I20260809 01:55:55.804384 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.796020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.3.220.62:35731)
0809 01:55:55.796282 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.796286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.796354 (+ 68us) server_negotiation.cc:407] Connection header received
0809 01:55:55.796501 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.796506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.796564 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.796672 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:55.797655 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.798171 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.799044 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.799279 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.799388 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.800693 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:55.800696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:55.800699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:55.800737 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:55.802615 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.803029 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.803036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.803038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.803121 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.803412 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.803417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.803418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.803702 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:55.803916 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.804118 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.804238 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
I20260809 01:55:55.804406 27746 heartbeater.cc:511] Master 127.3.220.62:35731 requested a full tablet report, sending...
I20260809 01:55:55.805465 27093 ts_manager.cc:194] Registered new tserver with Master: 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243)
I20260809 01:55:55.806062 27093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:39679
I20260809 01:55:55.806273 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:55:55.810053 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.807124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.3.220.62:35731)
0809 01:55:55.807255 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.807259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.807274 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:55:55.807344 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.807346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.807445 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.807550 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:55:55.807752 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.808343 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.809106 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.809317 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.809389 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.809671 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:55:55.809786 (+ 115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 01:55:55.809977 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60236: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 01:55:55.810142 27754 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60236: token expired
W20260809 01:55:55.810482 26988 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.220.62:35731)
I20260809 01:55:55.816993 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.810836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.3.220.62:35731)
0809 01:55:55.810947 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.810950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.810962 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:55:55.811045 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.811047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.811078 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.811141 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:55:55.811769 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.812229 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.812978 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.813169 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.813254 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.814512 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:55:55.814514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:55:55.814516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:55:55.814536 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:55:55.815735 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.816097 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.816099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.816101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.816168 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:55:55.816391 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:55:55.816393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:55:55.816394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:55:55.816536 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:55:55.816629 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.816815 (+ 186us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.816899 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 01:55:55.819201 27093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:55:55.822398 27093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:55:55.830585 27273 tablet_service.cc:1467] Processing CreateTablet for tablet 3327a10c473b4a3294677ff4a1bf56c8 (DEFAULT_TABLE table=test-table [id=2f15db6623084f79ae009687a6ce295c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:55.831338 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3327a10c473b4a3294677ff4a1bf56c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:55.834180 27212 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Bootstrap starting.
I20260809 01:55:55.835076 27212 tablet_bootstrap.cc:654] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:55.835438 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.829057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:44243 (local address 127.0.0.1:33136)
0809 01:55:55.829203 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:55.829216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:55.829324 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:55.830452 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:55.830455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:55.830475 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:55.830589 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.830595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.832721 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.832738 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.833695 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.833705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.834979 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.834982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.835076 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.835119 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 01:55:55.835201 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 01:55:55.835299 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":3}
I20260809 01:55:55.835451 27763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.829418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.3.220.3:44243)
0809 01:55:55.829857 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.829876 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.829901 (+ 25us) server_negotiation.cc:407] Connection header received
0809 01:55:55.830012 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.830020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.830268 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.830993 (+ 725us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:55.831537 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.832628 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.833825 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.834857 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.834980 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.835144 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.835229 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.835320 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":212,"threads_started":1}
I20260809 01:55:55.835889 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.829457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:34267 (local address 127.0.0.1:35796)
0809 01:55:55.830200 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:55.830214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:55.830317 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:55.830881 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:55.830884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:55.830902 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:55.831009 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.831019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.832668 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.832675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.834127 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.834137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.835423 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.835428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.835561 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.835589 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 01:55:55.835674 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 01:55:55.835760 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":220,"thread_start_us":157,"threads_started":1}
I20260809 01:55:55.836404 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.829744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.3.220.2:34267)
0809 01:55:55.830183 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.830194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.830315 (+ 121us) server_negotiation.cc:407] Connection header received
0809 01:55:55.830461 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.830478 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.830693 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.830884 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:55.831237 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.832534 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.834268 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.835085 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.835199 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.835346 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.836177 (+ 831us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.836269 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":169,"threads_started":1}
I20260809 01:55:55.836661 27212 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: No bootstrap required, opened a new log
I20260809 01:55:55.836755 27212 ts_tablet_manager.cc:1397] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 01:55:55.837213 27212 raft_consensus.cc:348] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.837340 27212 raft_consensus.cc:374] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:55.837379 27212 raft_consensus.cc:729] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4937de0d510453ba2922a146552986c, State: Initialized, Role: FOLLOWER
I20260809 01:55:55.837841 27212 consensus_queue.cc:260] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.838357 27212 ts_tablet_manager.cc:1428] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 01:55:55.838845 27647 tablet_service.cc:1467] Processing CreateTablet for tablet 3327a10c473b4a3294677ff4a1bf56c8 (DEFAULT_TABLE table=test-table [id=2f15db6623084f79ae009687a6ce295c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:55.839249 27470 tablet_service.cc:1467] Processing CreateTablet for tablet 3327a10c473b4a3294677ff4a1bf56c8 (DEFAULT_TABLE table=test-table [id=2f15db6623084f79ae009687a6ce295c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:55:55.840014 27647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3327a10c473b4a3294677ff4a1bf56c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:55.840242 27470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3327a10c473b4a3294677ff4a1bf56c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:55:55.846644 27766 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: Bootstrap starting.
I20260809 01:55:55.847173 27767 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Bootstrap starting.
I20260809 01:55:55.849021 27766 tablet_bootstrap.cc:654] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:55.849108 27767 tablet_bootstrap.cc:654] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Neither blocks nor log segments found. Creating new log.
I20260809 01:55:55.849871 27766 log.cc:824] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:55.849895 27767 log.cc:824] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Log is configured to *not* fsync() on all Append() calls
I20260809 01:55:55.851631 27767 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: No bootstrap required, opened a new log
I20260809 01:55:55.851763 27766 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: No bootstrap required, opened a new log
I20260809 01:55:55.851828 27767 ts_tablet_manager.cc:1397] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 01:55:55.851981 27766 ts_tablet_manager.cc:1397] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:55:55.854995 27767 raft_consensus.cc:348] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.855036 27766 raft_consensus.cc:348] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.855266 27767 raft_consensus.cc:374] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:55.855259 27766 raft_consensus.cc:374] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:55:55.855350 27766 raft_consensus.cc:729] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 711c58ba94c24ee4a0391f7409d89a8f, State: Initialized, Role: FOLLOWER
I20260809 01:55:55.855360 27767 raft_consensus.cc:729] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b18d6d3e6146df8386d501a3627c06, State: Initialized, Role: FOLLOWER
I20260809 01:55:55.855829 27767 consensus_queue.cc:260] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.855829 27766 consensus_queue.cc:260] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.856483 27746 heartbeater.cc:503] Master 127.3.220.62:35731 was elected leader, sending a full tablet report...
I20260809 01:55:55.856485 27557 heartbeater.cc:503] Master 127.3.220.62:35731 was elected leader, sending a full tablet report...
I20260809 01:55:55.857843 27767 ts_tablet_manager.cc:1428] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:55:55.857844 27766 ts_tablet_manager.cc:1428] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
W20260809 01:55:55.858022 27560 tablet.cc:2369] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 01:55:55.902544 27370 tablet.cc:2369] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:55:55.917996 27376 raft_consensus.cc:482] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:55:55.918319 27376 raft_consensus.cc:504] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.919471 27376 leader_election.cc:288] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243), 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
I20260809 01:55:55.924607 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.920301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:60419 (local address 127.3.220.2:34267)
0809 01:55:55.920434 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.920438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.920486 (+ 48us) server_negotiation.cc:407] Connection header received
0809 01:55:55.920646 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.920648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.920689 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.920792 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:55.921021 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.921771 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.923241 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.923844 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.923957 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.924016 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.924392 (+ 376us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.924461 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:55:55.924607 27763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.919905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:57677 (local address 127.3.220.3:44243)
0809 01:55:55.920076 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.920081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.920116 (+ 35us) server_negotiation.cc:407] Connection header received
0809 01:55:55.920276 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.920279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.920335 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.920436 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:55:55.920784 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.921567 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.923241 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.923844 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.923956 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.924016 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.924392 (+ 376us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.924461 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:55:55.924664 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.919771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:44243 (local address 127.3.220.1:57677)
0809 01:55:55.919921 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:55.919938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:55.920061 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:55.920451 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:55.920457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:55.920482 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:55.920652 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.920664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.921704 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.921706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.923100 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.923112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.924012 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.924019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.924191 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.924275 (+ 84us) client_negotiation.cc:769] Sending connection context
0809 01:55:55.924365 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 01:55:55.924453 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":122,"spinlock_wait_cycles":4096}
I20260809 01:55:55.924670 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.920078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:34267 (local address 127.3.220.1:60419)
0809 01:55:55.920407 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:55.920424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:55:55.920546 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:55:55.920814 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:55:55.920816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:55:55.920831 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:55:55.920920 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.920924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.921876 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.921879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.923100 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:55:55.923112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.924025 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.924029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:55:55.924197 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.924270 (+ 73us) client_negotiation.cc:769] Sending connection context
0809 01:55:55.924365 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 01:55:55.924453 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":56,"spinlock_wait_cycles":768,"thread_start_us":143,"threads_started":1}
I20260809 01:55:55.925463 27498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3327a10c473b4a3294677ff4a1bf56c8" candidate_uuid: "f4937de0d510453ba2922a146552986c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b18d6d3e6146df8386d501a3627c06" is_pre_election: true
I20260809 01:55:55.925464 27680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3327a10c473b4a3294677ff4a1bf56c8" candidate_uuid: "f4937de0d510453ba2922a146552986c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "711c58ba94c24ee4a0391f7409d89a8f" is_pre_election: true
I20260809 01:55:55.925836 27498 raft_consensus.cc:2454] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4937de0d510453ba2922a146552986c in term 0.
I20260809 01:55:55.925850 27680 raft_consensus.cc:2454] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4937de0d510453ba2922a146552986c in term 0.
I20260809 01:55:55.926371 27204 leader_election.cc:302] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 711c58ba94c24ee4a0391f7409d89a8f, f4937de0d510453ba2922a146552986c; no voters:
I20260809 01:55:55.926684 27376 raft_consensus.cc:2781] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:55:55.926808 27376 raft_consensus.cc:482] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:55:55.926877 27376 raft_consensus.cc:3037] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:55.927960 27376 raft_consensus.cc:504] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.928304 27376 leader_election.cc:288] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 1 election: Requested vote from peers 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243), 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
I20260809 01:55:55.928705 27680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3327a10c473b4a3294677ff4a1bf56c8" candidate_uuid: "f4937de0d510453ba2922a146552986c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "711c58ba94c24ee4a0391f7409d89a8f"
I20260809 01:55:55.928768 27498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3327a10c473b4a3294677ff4a1bf56c8" candidate_uuid: "f4937de0d510453ba2922a146552986c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b18d6d3e6146df8386d501a3627c06"
I20260809 01:55:55.928879 27680 raft_consensus.cc:3037] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:55.928898 27498 raft_consensus.cc:3037] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:55:55.930052 27498 raft_consensus.cc:2454] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4937de0d510453ba2922a146552986c in term 1.
I20260809 01:55:55.930317 27680 raft_consensus.cc:2454] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4937de0d510453ba2922a146552986c in term 1.
I20260809 01:55:55.930380 27204 leader_election.cc:302] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b18d6d3e6146df8386d501a3627c06, f4937de0d510453ba2922a146552986c; no voters:
I20260809 01:55:55.930558 27376 raft_consensus.cc:2781] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:55:55.930794 27376 raft_consensus.cc:686] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 LEADER]: Becoming Leader. State: Replica: f4937de0d510453ba2922a146552986c, State: Running, Role: LEADER
I20260809 01:55:55.931020 27376 consensus_queue.cc:237] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:55:55.934022 27091 catalog_manager.cc:5495] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c reported cstate change: term changed from 0 to 1, leader changed from <none> to f4937de0d510453ba2922a146552986c (127.3.220.1). New cstate: current_term: 1 leader_uuid: "f4937de0d510453ba2922a146552986c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } health_report { overall_health: HEALTHY } } }
I20260809 01:55:55.959363 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:55.955498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.3.220.1:42651)
0809 01:55:55.955736 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 01:55:55.955740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:55:55.955753 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:55:55.955834 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:55:55.955836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:55:55.955900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:55:55.956003 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:55:55.956294 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.956854 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.957892 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:55:55.958147 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:55:55.958235 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:55:55.958496 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:55:55.958963 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:55:55.959052 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 01:55:55.959183 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 01:55:55.959239 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260809 01:55:55.963542 27680 raft_consensus.cc:1261] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 1 FOLLOWER]: Refusing update from remote peer f4937de0d510453ba2922a146552986c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:55.963543 27498 raft_consensus.cc:1261] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 1 FOLLOWER]: Refusing update from remote peer f4937de0d510453ba2922a146552986c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:55:55.964246 27376 consensus_queue.cc:1035] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:55.964421 27773 consensus_queue.cc:1035] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [LEADER]: Connected to new peer: Peer: permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:55:55.971609 27778 mvcc.cc:205] Tried to move back new op lower bound from 7316441317220499456 to 7316441317095837696. Current Snapshot: MvccSnapshot[applied={T|T < 7316441317220499456}]
I20260809 01:55:55.972512 27782 mvcc.cc:205] Tried to move back new op lower bound from 7316441317220499456 to 7316441317095837696. Current Snapshot: MvccSnapshot[applied={T|T < 7316441317220499456}]
I20260809 01:55:55.981789 27090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:55.981982 27090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:55:55.984630 27090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Sending DeleteTablet for 3 replicas of tablet 3327a10c473b4a3294677ff4a1bf56c8
I20260809 01:55:55.985489 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 27146
I20260809 01:55:55.985459 27647 tablet_service.cc:1514] Processing DeleteTablet for tablet 3327a10c473b4a3294677ff4a1bf56c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:55 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:33136
I20260809 01:55:55.985739 27470 tablet_service.cc:1514] Processing DeleteTablet for tablet 3327a10c473b4a3294677ff4a1bf56c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:55:55 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:35796
I20260809 01:55:55.991453 27790 tablet_replica.cc:330] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: stopping tablet replica
I20260809 01:55:55.991850 27791 tablet_replica.cc:330] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: stopping tablet replica
I20260809 01:55:55.991884 27790 raft_consensus.cc:2229] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:55.992084 27791 raft_consensus.cc:2229] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:55:55.992271 27790 raft_consensus.cc:2258] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:55.992389 27791 raft_consensus.cc:2258] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:55:55.994046 27791 ts_tablet_manager.cc:1905] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:55.994163 27790 ts_tablet_manager.cc:1905] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:55:55.996439 27791 ts_tablet_manager.cc:1918] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:55.996570 27791 log.cc:1196] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/3327a10c473b4a3294677ff4a1bf56c8
I20260809 01:55:55.996951 27791 ts_tablet_manager.cc:1939] T 3327a10c473b4a3294677ff4a1bf56c8 P 35b18d6d3e6146df8386d501a3627c06: Deleting consensus metadata
I20260809 01:55:55.997689 27790 ts_tablet_manager.cc:1918] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:55:55.997792 27790 log.cc:1196] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/3327a10c473b4a3294677ff4a1bf56c8
I20260809 01:55:55.998194 27790 ts_tablet_manager.cc:1939] T 3327a10c473b4a3294677ff4a1bf56c8 P 711c58ba94c24ee4a0391f7409d89a8f: Deleting consensus metadata
I20260809 01:55:55.998320 27071 catalog_manager.cc:4841] TS 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267): tablet 3327a10c473b4a3294677ff4a1bf56c8 (table test-table [id=2f15db6623084f79ae009687a6ce295c]) successfully deleted
I20260809 01:55:55.999549 27071 catalog_manager.cc:4841] TS 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243): tablet 3327a10c473b4a3294677ff4a1bf56c8 (table test-table [id=2f15db6623084f79ae009687a6ce295c]) successfully deleted
W20260809 01:55:56.001428 27071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:42651 (error 108)
I20260809 01:55:56.002355 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 27373
I20260809 01:55:56.002656 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:55:56.002294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:42651 (local address 127.0.0.1:33376)
0809 01:55:56.002457 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 01:55:56.002544 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:42651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 01:55:56.009683 27071 catalog_manager.cc:4584] TS f4937de0d510453ba2922a146552986c (127.3.220.1:42651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3327a10c473b4a3294677ff4a1bf56c8: Network error: Client connection negotiation failed: client connection to 127.3.220.1:42651: connect: Connection refused (error 111)
I20260809 01:55:56.015312 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 27561
I20260809 01:55:56.030486 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 27045
I20260809 01:56:16.042490 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35731
--webserver_interface=127.3.220.62
--webserver_port=35477
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:16.160526 27794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:16.160799 27794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:16.160841 27794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:16.164968 27794 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 01:56:16.165032 27794 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 01:56:16.165048 27794 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 01:56:16.165061 27794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:16.165081 27794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:16.165129 27794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:16.165166 27794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:16.165186 27794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:16.170676 27794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:16.171854 27794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:16.172968 27794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:56:16.183158 27794 server_base.cc:1034] running on GCE node
W20260809 01:56:16.183108 27803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:16.183202 27802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:16.183106 27807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:16.184017 27794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:16.185096 27794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:16.186293 27794 hybrid_clock.cc:648] HybridClock initialized: now 1786240576186251 us; error 48 us; skew 500 ppm
Aug 09 01:56:16 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240576, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:16.192677 27794 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:56:16.193668 27794 webserver.cc:460] Webserver started at http://127.3.220.62:35477/ using document root <none> and password file <none>
I20260809 01:56:16.194223 27794 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:16.194288 27794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:16.198441 27794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260809 01:56:16.200649 27816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:16.202018 27794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:56:16.202133 27794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "13b011436ec644288b1ccac55b6c19f5"
format_stamp: "Formatted at 2026-08-09 01:55:33 on dist-test-slave-gkw7"
server_key: "f3e12311f65e800d7821defee66b392b"
server_key_iv: "c52711d4f0e123181a0973e63bf84097"
server_key_version: "encryptionkey@0"
I20260809 01:56:16.202682 27794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:16.216462 27794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:16.219959 27794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:16.220122 27794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:16.227968 27794 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:35731
I20260809 01:56:16.228332 27895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:35731 every 8 connection(s)
I20260809 01:56:16.229115 27794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:16.230851 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 27794
I20260809 01:56:16.231349 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:42651
--local_ip_for_outbound_sockets=127.3.220.1
--tserver_master_addrs=127.3.220.62:35731
--webserver_port=41339
--webserver_interface=127.3.220.1
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:56:16.233651 27899 sys_catalog.cc:263] Verifying existing consensus state
I20260809 01:56:16.235332 27899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Bootstrap starting.
I20260809 01:56:16.253302 27899 log.cc:824] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:16.264571 27899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:56:16.265015 27899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Bootstrap complete.
I20260809 01:56:16.269204 27899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:56:16.269713 27899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b011436ec644288b1ccac55b6c19f5, State: Initialized, Role: FOLLOWER
I20260809 01:56:16.270215 27899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:56:16.270363 27899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:16.270457 27899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:16.270574 27899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 2 FOLLOWER]: Advancing to term 3
I20260809 01:56:16.272468 27899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:56:16.272847 27899 leader_election.cc:302] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13b011436ec644288b1ccac55b6c19f5; no voters:
I20260809 01:56:16.273224 27899 leader_election.cc:288] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 01:56:16.273347 27903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 01:56:16.273628 27903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [term 3 LEADER]: Becoming Leader. State: Replica: 13b011436ec644288b1ccac55b6c19f5, State: Running, Role: LEADER
I20260809 01:56:16.273993 27903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } }
I20260809 01:56:16.274477 27899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:16.275511 27905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b011436ec644288b1ccac55b6c19f5. Latest consensus state: current_term: 3 leader_uuid: "13b011436ec644288b1ccac55b6c19f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } } }
I20260809 01:56:16.275656 27905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:16.275906 27904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "13b011436ec644288b1ccac55b6c19f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b011436ec644288b1ccac55b6c19f5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 35731 } } }
I20260809 01:56:16.275982 27904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:16.276012 27915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:16.281471 27915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:16.282250 27915 catalog_manager.cc:1259] Loaded cluster ID: 73245ac378a6474493beac282a63d32c
I20260809 01:56:16.282313 27915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:16.284716 27915 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:16.285444 27915 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Loaded TSK: 0
I20260809 01:56:16.285555 27915 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13b011436ec644288b1ccac55b6c19f5: Loaded TSK: 1
F20260809 01:56:16.286331 27915 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f369bd389ad google::LogMessage::Fail() at ??:0
@ 0x7f369bd3b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f369bd384de google::LogMessage::Flush() at ??:0
@ 0x7f369bd39a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f369e9fc18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f369e9f4f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f36a669d7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f36a667799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f36a6678178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f36a66b9df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f36a66d882b std::function<>::operator()() at ??:0
@ 0x7f36a6677b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f36a6678f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f36a6675579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f36a66b9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f369df8a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f369df8ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f369df8c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f369df7b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f369f5bf6db start_thread at ??:0
@ 0x7f369a83661f clone at ??:0
W20260809 01:56:16.388028 27901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:16.388447 27901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:16.388532 27901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:16.395139 27901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:16.395241 27901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:16.395350 27901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:56:16.403796 27901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:42651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=41339
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:16.405376 27901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:16.406840 27901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:16.418552 27962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:16.418552 27961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:16.418597 27968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:16.419162 27901 server_base.cc:1034] running on GCE node
I20260809 01:56:16.419582 27901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:16.420264 27901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:16.421499 27901 hybrid_clock.cc:648] HybridClock initialized: now 1786240576421468 us; error 52 us; skew 500 ppm
Aug 09 01:56:16 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240576, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:16.424458 27901 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:56:16.425410 27901 webserver.cc:460] Webserver started at http://127.3.220.1:41339/ using document root <none> and password file <none>
I20260809 01:56:16.426019 27901 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:16.426084 27901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:16.430274 27901 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 01:56:16.432682 27975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:16.434180 27901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:56:16.434316 27901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "f4937de0d510453ba2922a146552986c"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "fe65dfc2a5089320f7c014acfd6bbd4a"
server_key_iv: "2088a833bd9a0f1fa1f9afb5be5cc852"
server_key_version: "encryptionkey@0"
I20260809 01:56:16.434866 27901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:16.452011 27901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:16.455390 27901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:16.455607 27901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:16.456228 27901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:16.457381 27989 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 01:56:16.463079 27901 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 01:56:16.463186 27901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260809 01:56:16.463274 27901 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 01:56:16.464987 27901 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 01:56:16.465059 27901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 01:56:16.465377 27989 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Bootstrap starting.
I20260809 01:56:16.476166 27901 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:42651
I20260809 01:56:16.476334 28148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:42651 every 8 connection(s)
I20260809 01:56:16.477276 27901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:56:16.479990 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 27901
I20260809 01:56:16.480513 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:34267
--local_ip_for_outbound_sockets=127.3.220.2
--tserver_master_addrs=127.3.220.62:35731
--webserver_port=39269
--webserver_interface=127.3.220.2
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:56:16.488466 27989 log.cc:824] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:16.498504 27989 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 01:56:16.499032 27989 tablet_bootstrap.cc:492] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Bootstrap complete.
I20260809 01:56:16.500561 27989 ts_tablet_manager.cc:1397] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Time spent bootstrapping tablet: real 0.035s user 0.017s sys 0.015s
I20260809 01:56:16.504276 27989 raft_consensus.cc:348] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:56:16.504896 27989 raft_consensus.cc:729] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4937de0d510453ba2922a146552986c, State: Initialized, Role: FOLLOWER
I20260809 01:56:16.505505 27989 consensus_queue.cc:260] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:56:16.506740 27989 ts_tablet_manager.cc:1428] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20260809 01:56:16.549832 28159 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:56:16.479364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:44021)
0809 01:56:16.480550 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.480615 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:16.481913 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:16.548812 (+ 66899us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":149,"threads_started":1}
W20260809 01:56:16.550477 28153 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:35731 (0 consecutive failures): Network error: Failed to ping master at 127.3.220.62:35731: Client connection negotiation failed: client connection to 127.3.220.62:35731: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 01:56:16.551295 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:16.550993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:33307)
0809 01:56:16.551145 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.551230 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 01:56:16.551995 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:16.551777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:42753)
0809 01:56:16.551886 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.551937 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 01:56:16.552206 28153 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 01:56:16.602121 28162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:16.602381 28162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:16.602452 28162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:16.606377 28162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:16.606431 28162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:16.606527 28162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:56:16.611680 28162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:34267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=39269
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:16.612761 28162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:16.613850 28162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:16.623922 28174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:16.623972 28170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:16.623943 28171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:16.623993 28162 server_base.cc:1034] running on GCE node
I20260809 01:56:16.624596 28162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:16.625241 28162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:16.626405 28162 hybrid_clock.cc:648] HybridClock initialized: now 1786240576626370 us; error 48 us; skew 500 ppm
Aug 09 01:56:16 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240576, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:16.629166 28162 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:56:16.630117 28162 webserver.cc:460] Webserver started at http://127.3.220.2:39269/ using document root <none> and password file <none>
I20260809 01:56:16.630641 28162 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:16.630710 28162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:16.634868 28162 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:56:16.637056 28187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:16.638428 28162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:56:16.638557 28162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "35b18d6d3e6146df8386d501a3627c06"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "659c65619a12f69d9d20d27e1a3988bc"
server_key_iv: "01017d734fb37e67e4174a1f2f034b99"
server_key_version: "encryptionkey@0"
I20260809 01:56:16.639199 28162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:16.662992 28162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:16.666433 28162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:16.666651 28162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:16.667299 28162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:16.668181 28162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:16.668232 28162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:16.668309 28162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:16.668352 28162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:16.677825 28162 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:34267
I20260809 01:56:16.677853 28349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:34267 every 8 connection(s)
I20260809 01:56:16.678850 28162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:56:16.680462 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 28162
I20260809 01:56:16.680914 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:44243
--local_ip_for_outbound_sockets=127.3.220.3
--tserver_master_addrs=127.3.220.62:35731
--webserver_port=37029
--webserver_interface=127.3.220.3
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:56:16.683058 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:16.681070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:52401)
0809 01:56:16.681623 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.681762 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":153,"threads_started":1}
W20260809 01:56:16.683683 28350 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:35731 (0 consecutive failures): Network error: Failed to ping master at 127.3.220.62:35731: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
I20260809 01:56:16.684362 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:16.684120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:58267)
0809 01:56:16.684242 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.684294 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 01:56:16.685084 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:16.684876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:54783)
0809 01:56:16.684986 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.685037 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 01:56:16.685281 28350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 01:56:16.798884 28354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:16.799172 28354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:16.799247 28354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:16.803264 28354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:16.803351 28354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:16.803442 28354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:56:16.808746 28354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:44243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=37029
--tserver_master_addrs=127.3.220.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:16.809865 28354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:16.810932 28354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:16.821126 28367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:16.821260 28363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:16.821142 28364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:16.821754 28354 server_base.cc:1034] running on GCE node
I20260809 01:56:16.822180 28354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:16.822857 28354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:16.824052 28354 hybrid_clock.cc:648] HybridClock initialized: now 1786240576824033 us; error 29 us; skew 500 ppm
Aug 09 01:56:16 dist-test-slave-gkw7 krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240576, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:16.827323 28354 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:56:16.828372 28354 webserver.cc:460] Webserver started at http://127.3.220.3:37029/ using document root <none> and password file <none>
I20260809 01:56:16.829015 28354 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:16.829084 28354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:16.833400 28354 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:56:16.835785 28377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:16.837121 28354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:56:16.837287 28354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "711c58ba94c24ee4a0391f7409d89a8f"
format_stamp: "Formatted at 2026-08-09 01:55:34 on dist-test-slave-gkw7"
server_key: "794e06c06c51f6f7c1294a98be8730f4"
server_key_iv: "24769ca62c531bbdabe7713c9d166447"
server_key_version: "encryptionkey@0"
I20260809 01:56:16.837987 28354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:16.857075 28354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:16.860877 28354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:16.861116 28354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:16.861884 28354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:16.862800 28354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:16.862879 28354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:16.862957 28354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:16.863013 28354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:16.872839 28354 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:44243
I20260809 01:56:16.872866 28529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:44243 every 8 connection(s)
I20260809 01:56:16.873983 28354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:56:16.878115 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:16.876201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:36067)
0809 01:56:16.876665 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.876785 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
W20260809 01:56:16.878630 28531 heartbeater.cc:644] Failed to heartbeat to 127.3.220.62:35731 (0 consecutive failures): Network error: Failed to ping master at 127.3.220.62:35731: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
I20260809 01:56:16.879243 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:16.879029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:55489)
0809 01:56:16.879136 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.879187 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260809 01:56:16.879477 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 28354
I20260809 01:56:16.879971 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:16.879739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:45081)
0809 01:56:16.879859 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:16.879919 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 01:56:16.880215 28531 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 01:56:17.553345 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:17.552891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:43739)
0809 01:56:17.553150 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:17.553208 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260809 01:56:17.686429 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:17.685971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:46897)
0809 01:56:17.686258 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:17.686317 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":109,"threads_started":1}
I20260809 01:56:17.881460 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:17.880968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:57195)
0809 01:56:17.881288 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:17.881353 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":81,"threads_started":1}
I20260809 01:56:18.378590 28538 raft_consensus.cc:482] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:18.378814 28538 raft_consensus.cc:504] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:56:18.379997 28538 leader_election.cc:288] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243), 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
Aug 09 01:56:18 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240576, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.2@KRBTEST.COM
Aug 09 01:56:18 dist-test-slave-gkw7 krb5kdc[26271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240576, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.3@KRBTEST.COM
I20260809 01:56:18.394464 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:18.381050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:48277 (local address 127.3.220.2:34267)
0809 01:56:18.381407 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:18.381418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:18.381451 (+ 33us) server_negotiation.cc:407] Connection header received
0809 01:56:18.382154 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:18.382160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:18.382470 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:18.382763 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:18.383989 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:18.384980 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:18.386251 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:18.386725 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:18.386915 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:18.389649 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:18.389656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:18.389674 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:18.389711 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:18.391602 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:18.392274 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:18.392279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:18.392283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:18.392400 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:18.392758 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:18.392763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:18.392765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:18.393277 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:18.393424 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:18.394007 (+ 583us) server_negotiation.cc:299] Negotiation successful
0809 01:56:18.394272 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":145,"threads_started":1}
I20260809 01:56:18.394505 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:18.380663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:34267 (local address 127.3.220.1:48277)
0809 01:56:18.380974 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:18.380997 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:18.381079 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:18.382763 (+ 1684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:18.382768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:18.383351 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:18.383792 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:18.383807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:18.385128 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:18.385131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:18.386114 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:18.386125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:18.386874 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:18.386879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:18.387079 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:18.387829 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:18.387842 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:18.389354 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:18.391734 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:18.391740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:18.391755 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:18.392140 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:18.392504 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:18.392508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:18.392510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:18.392655 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:18.393396 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:18.393403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:18.393695 (+ 292us) client_negotiation.cc:769] Sending connection context
0809 01:56:18.393888 (+ 193us) client_negotiation.cc:240] Negotiation successful
0809 01:56:18.394212 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":55,"thread_start_us":48,"threads_started":1}
I20260809 01:56:18.394539 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:18.380258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:44243 (local address 127.3.220.1:59665)
0809 01:56:18.380564 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:18.380579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:18.380745 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:18.382761 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:18.382768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:18.383383 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:18.383793 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:18.383807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:18.385188 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:18.385192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:18.386118 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:18.386128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:18.386867 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:18.386871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:18.387092 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:18.387760 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:18.387777 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:18.389817 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:18.392215 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:18.392219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:18.392221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:18.392516 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:18.392903 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:18.392908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:18.392910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:18.393018 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:18.393767 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:18.393770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:18.393883 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 01:56:18.394006 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 01:56:18.394320 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":259,"spinlock_wait_cycles":9728,"thread_start_us":102,"threads_started":1}
I20260809 01:56:18.394701 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:18.380474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:59665 (local address 127.3.220.3:44243)
0809 01:56:18.380843 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:18.380874 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:18.380953 (+ 79us) server_negotiation.cc:407] Connection header received
0809 01:56:18.381950 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:18.381963 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:18.382365 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:18.382722 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:18.383989 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:18.385030 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:18.386262 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:18.386732 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:18.386914 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:18.390002 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:18.390007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:18.390018 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:18.390070 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:18.392043 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:18.392650 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:18.392658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:18.392663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:18.392789 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:18.393164 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:18.393168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:18.393170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:18.393643 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:18.393785 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:18.394192 (+ 407us) server_negotiation.cc:299] Negotiation successful
0809 01:56:18.394503 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
W20260809 01:56:18.397320 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
W20260809 01:56:18.397765 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
I20260809 01:56:18.397858 27979 leader_election.cc:302] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4937de0d510453ba2922a146552986c; no voters: 35b18d6d3e6146df8386d501a3627c06, 711c58ba94c24ee4a0391f7409d89a8f
I20260809 01:56:18.398151 28538 raft_consensus.cc:2726] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:56:18.554644 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:18.554320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:54783)
0809 01:56:18.554482 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:18.554556 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 01:56:18.687794 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:18.687323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:45001)
0809 01:56:18.687600 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:18.687663 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260809 01:56:18.882798 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:18.882334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:50717)
0809 01:56:18.882615 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:18.882673 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260809 01:56:19.555820 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:19.555387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:38479)
0809 01:56:19.555658 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:19.555716 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260809 01:56:19.688966 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:19.688553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:48099)
0809 01:56:19.688816 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:19.688877 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260809 01:56:19.883905 28551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:19.883498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:47253)
0809 01:56:19.883746 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:19.883803 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260809 01:56:19.990068 28552 raft_consensus.cc:482] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:19.990211 28552 raft_consensus.cc:504] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:56:19.990717 28552 leader_election.cc:288] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243), 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
W20260809 01:56:19.991677 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
W20260809 01:56:19.991847 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
I20260809 01:56:19.991912 27979 leader_election.cc:302] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4937de0d510453ba2922a146552986c; no voters: 35b18d6d3e6146df8386d501a3627c06, 711c58ba94c24ee4a0391f7409d89a8f
I20260809 01:56:19.992044 28552 raft_consensus.cc:2726] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:56:20.557279 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:20.556664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:41493)
0809 01:56:20.557085 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:20.557164 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":77,"threads_started":1}
I20260809 01:56:20.690408 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:20.689937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:57439)
0809 01:56:20.690211 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:20.690268 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260809 01:56:20.885270 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:20.884800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:44957)
0809 01:56:20.885083 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:20.885145 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260809 01:56:21.558743 28557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:21.558315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:42447)
0809 01:56:21.558592 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:21.558645 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260809 01:56:21.691735 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:21.691266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:52691)
0809 01:56:21.691572 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:21.691636 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":111,"threads_started":1}
I20260809 01:56:21.886723 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:21.886218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:54943)
0809 01:56:21.886534 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:21.886604 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
I20260809 01:56:22.479099 28560 raft_consensus.cc:482] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:22.479229 28560 raft_consensus.cc:504] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:56:22.479686 28560 leader_election.cc:288] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243), 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
W20260809 01:56:22.480595 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
W20260809 01:56:22.480789 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
I20260809 01:56:22.480855 27979 leader_election.cc:302] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4937de0d510453ba2922a146552986c; no voters: 35b18d6d3e6146df8386d501a3627c06, 711c58ba94c24ee4a0391f7409d89a8f
I20260809 01:56:22.481053 28560 raft_consensus.cc:2726] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:56:22.560070 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:22.559619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:42227)
0809 01:56:22.559909 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:22.559969 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
I20260809 01:56:22.693300 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:22.692749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:40317)
0809 01:56:22.693059 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:22.693192 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260809 01:56:22.888144 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:22.887678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:43301)
0809 01:56:22.887980 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:22.888041 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260809 01:56:23.561352 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:23.560907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:52033)
0809 01:56:23.561186 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:23.561244 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260809 01:56:23.694581 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:23.694122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:52211)
0809 01:56:23.694422 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:23.694485 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":69,"threads_started":1}
I20260809 01:56:23.889581 28569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:23.889070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:40399)
0809 01:56:23.889379 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:23.889437 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":78,"threads_started":1}
I20260809 01:56:24.448470 28570 raft_consensus.cc:482] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:24.448611 28570 raft_consensus.cc:504] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:56:24.449113 28570 leader_election.cc:288] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243), 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
W20260809 01:56:24.450029 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
W20260809 01:56:24.450234 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
I20260809 01:56:24.450309 27979 leader_election.cc:302] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4937de0d510453ba2922a146552986c; no voters: 35b18d6d3e6146df8386d501a3627c06, 711c58ba94c24ee4a0391f7409d89a8f
I20260809 01:56:24.450455 28570 raft_consensus.cc:2726] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:56:24.562736 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:24.562212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:56359)
0809 01:56:24.562498 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:24.562597 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":88,"threads_started":1}
I20260809 01:56:24.695866 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:24.695433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:38415)
0809 01:56:24.695704 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:24.695763 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260809 01:56:24.890987 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:24.890474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:39529)
0809 01:56:24.890757 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:24.890829 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
I20260809 01:56:25.564092 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:25.563639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:48445)
0809 01:56:25.563928 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:25.563988 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20260809 01:56:25.697063 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:25.696614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:32917)
0809 01:56:25.696907 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:25.696965 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":108,"threads_started":1}
I20260809 01:56:25.892407 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:25.891898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:45129)
0809 01:56:25.892181 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:25.892262 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
I20260809 01:56:26.565697 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:26.565009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:53117)
0809 01:56:26.565468 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:26.565588 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":104,"threads_started":1}
I20260809 01:56:26.698453 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:26.698042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:58831)
0809 01:56:26.698295 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:26.698356 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260809 01:56:26.893749 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:26.893247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:57781)
0809 01:56:26.893579 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:26.893642 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260809 01:56:27.566985 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:27.566525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:37409)
0809 01:56:27.566828 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:27.566887 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260809 01:56:27.699815 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:27.699340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:38627)
0809 01:56:27.699645 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:27.699711 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
I20260809 01:56:27.895030 28587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:27.894546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:32821)
0809 01:56:27.894864 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:27.894928 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
I20260809 01:56:28.568907 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:28.568177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:33129)
0809 01:56:28.568616 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:28.568716 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":171,"threads_started":1}
I20260809 01:56:28.701668 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:28.700800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:34817)
0809 01:56:28.701271 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:28.701412 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":122,"threads_started":1}
I20260809 01:56:28.896653 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:28.896008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:33903)
0809 01:56:28.896437 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:28.896507 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":104,"threads_started":1}
I20260809 01:56:29.389461 28592 raft_consensus.cc:482] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:29.389752 28592 raft_consensus.cc:504] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "711c58ba94c24ee4a0391f7409d89a8f" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 44243 } } peers { permanent_uuid: "35b18d6d3e6146df8386d501a3627c06" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 34267 } } peers { permanent_uuid: "f4937de0d510453ba2922a146552986c" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 42651 } }
I20260809 01:56:29.390630 28592 leader_election.cc:288] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243), 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267)
W20260809 01:56:29.391844 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 711c58ba94c24ee4a0391f7409d89a8f (127.3.220.3:44243): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
W20260809 01:56:29.392077 27979 leader_election.cc:341] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35b18d6d3e6146df8386d501a3627c06 (127.3.220.2:34267): Not found: Tablet not found: 3327a10c473b4a3294677ff4a1bf56c8
I20260809 01:56:29.392177 27979 leader_election.cc:302] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4937de0d510453ba2922a146552986c; no voters: 35b18d6d3e6146df8386d501a3627c06, 711c58ba94c24ee4a0391f7409d89a8f
I20260809 01:56:29.392477 28592 raft_consensus.cc:2726] T 3327a10c473b4a3294677ff4a1bf56c8 P f4937de0d510453ba2922a146552986c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 01:56:29.570964 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:29.570236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:52555)
0809 01:56:29.570677 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:29.570775 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":169,"threads_started":1}
I20260809 01:56:29.703006 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:29.702589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:60143)
0809 01:56:29.702848 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:29.702906 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260809 01:56:29.898134 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:29.897691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:52443)
0809 01:56:29.897965 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:29.898031 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260809 01:56:30.572515 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:30.572068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:38061)
0809 01:56:30.572352 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:30.572412 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260809 01:56:30.704370 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:30.703875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:53945)
0809 01:56:30.704196 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:30.704267 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
I20260809 01:56:30.899402 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:30.898962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:44491)
0809 01:56:30.899228 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:30.899285 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260809 01:56:31.574040 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:31.573611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.1:37459)
0809 01:56:31.573880 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:31.573938 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":83,"threads_started":1}
I20260809 01:56:31.706059 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:31.705510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.2:34843)
0809 01:56:31.705898 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:31.705959 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":119,"threads_started":1}
I20260809 01:56:31.884001 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 27901
I20260809 01:56:31.900379 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 28162
I20260809 01:56:31.906008 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:31.900443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:35731 (local address 127.3.220.3:45435)
0809 01:56:31.905709 (+ 5266us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:31.905832 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.62:35731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5127,"thread_start_us":74,"threads_started":1}
I20260809 01:56:31.914645 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 28354
2026-08-09T01:56:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59451 ms)
[----------] 30 tests from SecurityITest (253708 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:31 dist-test-slave-gkw7 krb5kdc[28610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:31 dist-test-slave-gkw7 krb5kdc[28610](info): set up 2 sockets
Aug 09 01:56:31 dist-test-slave-gkw7 krb5kdc[28610](info): commencing operation
krb5kdc: starting...
W20260809 01:56:33.167330 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:33 dist-test-slave-gkw7 krb5kdc[28610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:56:33.309059 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:37457
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:37457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:33.423481 28627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:33.423750 28627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:33.423791 28627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:33.427835 28627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:33.427892 28627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:33.427911 28627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:33.427928 28627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:33.427943 28627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:33.433071 28627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:37457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:37457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:33.434116 28627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:33.435190 28627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:33.444299 28636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:33.444303 28635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:33.444278 28638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:33.444552 28627 server_base.cc:1034] running on GCE node
I20260809 01:56:33.445248 28627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:33.446455 28627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:33.447685 28627 hybrid_clock.cc:648] HybridClock initialized: now 1786240593447669 us; error 26 us; skew 500 ppm
Aug 09 01:56:33 dist-test-slave-gkw7 krb5kdc[28610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:33.450542 28627 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:56:33.451439 28627 webserver.cc:460] Webserver started at http://127.3.220.62:36329/ using document root <none> and password file <none>
I20260809 01:56:33.452024 28627 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:33.452128 28627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:33.452384 28627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:33.454546 28627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "87b4ae6528194e27b3f3c8429ae00f37"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "05a3e057fb58ed31ae1d1a2d09c141cf"
server_key_iv: "59adfcae82503b55d07723ca6a0634cb"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.455127 28627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "87b4ae6528194e27b3f3c8429ae00f37"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "05a3e057fb58ed31ae1d1a2d09c141cf"
server_key_iv: "59adfcae82503b55d07723ca6a0634cb"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.458963 28627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:56:33.461395 28647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:33.462651 28627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:33.462817 28627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "87b4ae6528194e27b3f3c8429ae00f37"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "05a3e057fb58ed31ae1d1a2d09c141cf"
server_key_iv: "59adfcae82503b55d07723ca6a0634cb"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.462940 28627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:33.484102 28627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:33.487298 28627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:33.487527 28627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:33.494711 28627 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:37457
I20260809 01:56:33.494752 28717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:37457 every 8 connection(s)
I20260809 01:56:33.495705 28627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:33.496127 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 28627
I20260809 01:56:33.496266 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:33.496562 3952 external_mini_cluster.cc:1428] Setting key 2f89ca7dd172c71b8437300723eb6be5
I20260809 01:56:33.498858 28718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:33.504676 28718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37: Bootstrap starting.
Aug 09 01:56:33 dist-test-slave-gkw7 krb5kdc[28610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:33.507177 28718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:33.508060 28718 log.cc:824] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:33.510102 28718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37: No bootstrap required, opened a new log
I20260809 01:56:33.511235 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:33.498010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.3.220.62:37457)
0809 01:56:33.498438 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:33.498461 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:33.498701 (+ 240us) server_negotiation.cc:407] Connection header received
0809 01:56:33.499547 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:33.499565 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:33.499907 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:33.500267 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:33.501113 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:33.502236 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:33.503135 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:33.503655 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:33.503852 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:33.506541 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:33.506546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:33.506560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:33.506587 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:33.508399 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:33.508852 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:33.508858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:33.508862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:33.508973 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:33.509273 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:33.509277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:33.509279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:33.509743 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:33.509946 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:33.510239 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 01:56:33.510537 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":139,"threads_started":1}
I20260809 01:56:33.513461 28718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b4ae6528194e27b3f3c8429ae00f37" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 37457 } }
I20260809 01:56:33.513803 28718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:33.513906 28718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b4ae6528194e27b3f3c8429ae00f37, State: Initialized, Role: FOLLOWER
I20260809 01:56:33.514341 28718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b4ae6528194e27b3f3c8429ae00f37" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 37457 } }
I20260809 01:56:33.514513 28718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:33.514608 28718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:33.514721 28718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:33.515648 28718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b4ae6528194e27b3f3c8429ae00f37" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 37457 } }
I20260809 01:56:33.516000 28718 leader_election.cc:302] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b4ae6528194e27b3f3c8429ae00f37; no voters:
I20260809 01:56:33.516358 28718 leader_election.cc:288] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:33.516489 28724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:33.516754 28724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [term 1 LEADER]: Becoming Leader. State: Replica: 87b4ae6528194e27b3f3c8429ae00f37, State: Running, Role: LEADER
I20260809 01:56:33.517129 28724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b4ae6528194e27b3f3c8429ae00f37" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 37457 } }
I20260809 01:56:33.517745 28718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:33.518944 28727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b4ae6528194e27b3f3c8429ae00f37. Latest consensus state: current_term: 1 leader_uuid: "87b4ae6528194e27b3f3c8429ae00f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b4ae6528194e27b3f3c8429ae00f37" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 37457 } } }
I20260809 01:56:33.519083 28727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:33.519397 28726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b4ae6528194e27b3f3c8429ae00f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b4ae6528194e27b3f3c8429ae00f37" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 37457 } } }
I20260809 01:56:33.519497 28726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:33.519495 28739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:33.522557 28739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:33.528162 28739 catalog_manager.cc:1347] Generated new cluster ID: 24df948fcb99493bbeca4cecc08b624f
I20260809 01:56:33.528244 28739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:33.546249 28739 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:33.547310 28739 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:33.551941 28739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37: Generated new TSK 0
I20260809 01:56:33.552868 28739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:56:33.594380 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:37457
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:33.709112 28758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:33.709399 28758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:33.709443 28758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:33.713429 28758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:33.713493 28758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:33.713631 28758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:56:33.719206 28758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:33.720297 28758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:33.721351 28758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:33.731381 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:33.731417 28768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:33.731388 28766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:33.731665 28758 server_base.cc:1034] running on GCE node
I20260809 01:56:33.732097 28758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:33.732668 28758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:33.733867 28758 hybrid_clock.cc:648] HybridClock initialized: now 1786240593733846 us; error 30 us; skew 500 ppm
Aug 09 01:56:33 dist-test-slave-gkw7 krb5kdc[28610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:33.736672 28758 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:56:33.737682 28758 webserver.cc:460] Webserver started at http://127.3.220.1:43029/ using document root <none> and password file <none>
I20260809 01:56:33.738252 28758 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:33.738318 28758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:33.738566 28758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:33.740592 28758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "f417a7d2690d43ad85bb859c1b9d37c9"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "b2fce11c98d348b0da49ad119b84d0d2"
server_key_iv: "4573bd708c8a119279b159a1307de64c"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.741173 28758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "f417a7d2690d43ad85bb859c1b9d37c9"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "b2fce11c98d348b0da49ad119b84d0d2"
server_key_iv: "4573bd708c8a119279b159a1307de64c"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.744874 28758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260809 01:56:33.747352 28784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:33.748515 28758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 01:56:33.748682 28758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "f417a7d2690d43ad85bb859c1b9d37c9"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "b2fce11c98d348b0da49ad119b84d0d2"
server_key_iv: "4573bd708c8a119279b159a1307de64c"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.748791 28758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:33.783522 28758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:33.786644 28758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:33.786870 28758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:33.787493 28758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:33.788347 28758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:33.788394 28758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:33.788470 28758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:33.788502 28758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:33.797652 28758 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:36441
I20260809 01:56:33.797675 28952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:36441 every 8 connection(s)
I20260809 01:56:33.798553 28758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:56:33.802090 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 28758
I20260809 01:56:33.802246 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:56:33.802557 3952 external_mini_cluster.cc:1428] Setting key 98d6cb36b2f9629af063873bb1aefaf8
Aug 09 01:56:33 dist-test-slave-gkw7 krb5kdc[28610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:33.813442 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:33.800420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:54295 (local address 127.3.220.62:37457)
0809 01:56:33.800564 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:33.800568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:33.801359 (+ 791us) server_negotiation.cc:407] Connection header received
0809 01:56:33.802826 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:33.802829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:33.802890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:33.803002 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:33.804529 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:33.805366 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:33.806633 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:33.806883 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:33.807002 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:33.809512 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:33.809516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:33.809519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:33.809571 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:33.811130 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:33.811796 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:33.811799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:33.811800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:33.811877 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:33.812298 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:33.812301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:33.812302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:33.812470 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:33.812546 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:33.813198 (+ 652us) server_negotiation.cc:299] Negotiation successful
0809 01:56:33.813320 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:56:33.814561 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:33.800731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:37457 (local address 127.3.220.1:54295)
0809 01:56:33.801186 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:33.801229 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:33.802560 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:33.803213 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:33.803221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:33.803709 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:33.804369 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:33.804385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:33.805498 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:33.805501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:33.806498 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:33.806508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:33.807047 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:33.807052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:33.807228 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:33.807845 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:33.807863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:33.809331 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:33.811283 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:33.811292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:33.811312 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:33.811689 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:33.811978 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:33.811983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:33.811986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:33.812189 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:33.812585 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:33.812591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:33.812931 (+ 340us) client_negotiation.cc:769] Sending connection context
0809 01:56:33.813177 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 01:56:33.813575 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260809 01:56:33.815785 28953 heartbeater.cc:347] Connected to a master server at 127.3.220.62:37457
I20260809 01:56:33.816043 28953 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:33.816581 28953 heartbeater.cc:511] Master 127.3.220.62:37457 requested a full tablet report, sending...
I20260809 01:56:33.818210 28668 ts_manager.cc:194] Registered new tserver with Master: f417a7d2690d43ad85bb859c1b9d37c9 (127.3.220.1:36441)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:56:33.819448 28668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:54295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:56:33.836402 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:37457
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:33.949514 28960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:33.949880 28960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:33.949970 28960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:33.954138 28960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:33.954216 28960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:33.954308 28960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:56:33.959435 28960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:33.960544 28960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:33.961683 28960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:33.973141 28967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:33.973219 28970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:33.973508 28960 server_base.cc:1034] running on GCE node
W20260809 01:56:33.973168 28968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:33.974139 28960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:33.976580 28960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:33.977808 28960 hybrid_clock.cc:648] HybridClock initialized: now 1786240593977766 us; error 63 us; skew 500 ppm
Aug 09 01:56:33 dist-test-slave-gkw7 krb5kdc[28610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:33.981290 28960 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:56:33.982620 28960 webserver.cc:460] Webserver started at http://127.3.220.2:44647/ using document root <none> and password file <none>
I20260809 01:56:33.983465 28960 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:33.983556 28960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:33.983806 28960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:33.987111 28960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "fe71788fc30b4f15c248ff80119d33b1"
server_key_iv: "ee71124df915459702ccfa9ebc25e057"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.987733 28960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "fe71788fc30b4f15c248ff80119d33b1"
server_key_iv: "ee71124df915459702ccfa9ebc25e057"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.993183 28960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 01:56:33.996124 28981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:33.997907 28960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 01:56:33.998049 28960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e"
format_stamp: "Formatted at 2026-08-09 01:56:33 on dist-test-slave-gkw7"
server_key: "fe71788fc30b4f15c248ff80119d33b1"
server_key_iv: "ee71124df915459702ccfa9ebc25e057"
server_key_version: "encryptionkey@0"
I20260809 01:56:33.998186 28960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:34.037741 28960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:34.041083 28960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:34.041292 28960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:34.041986 28960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:34.042795 28960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:34.042845 28960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:34.042886 28960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:34.042900 28960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:34.051551 28960 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:40745
I20260809 01:56:34.051599 29155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:40745 every 8 connection(s)
I20260809 01:56:34.052585 28960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:56:34.053339 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 28960
I20260809 01:56:34.053470 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:56:34.053808 3952 external_mini_cluster.cc:1428] Setting key d45b52a5e921653fe862d5aa3bb7199b
Aug 09 01:56:34 dist-test-slave-gkw7 krb5kdc[28610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:56:34.069259 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.054795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:48881 (local address 127.3.220.62:37457)
0809 01:56:34.054942 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.054947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.056032 (+ 1085us) server_negotiation.cc:407] Connection header received
0809 01:56:34.057952 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.057958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.058022 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.058119 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:34.059494 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.060022 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.061985 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.062206 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.062303 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.065208 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:34.065212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:34.065214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:34.065237 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:34.066991 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:34.067602 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:34.067608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:34.067609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:34.067722 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:34.068088 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:34.068094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:34.068098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:34.068335 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:34.068453 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.068961 (+ 508us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.069085 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:56:34.070273 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.055297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:37457 (local address 127.3.220.2:48881)
0809 01:56:34.055846 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.055911 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.057690 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.058309 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.058316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.058679 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:34.059329 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.059353 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.060168 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.060175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.061834 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.061868 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.062384 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.062391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.062642 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.063311 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:34.063335 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:34.065025 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:34.067132 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:34.067139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:34.067154 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:34.067417 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:34.067830 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:34.067834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:34.067836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:34.067983 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:34.068454 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:34.068463 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:34.068708 (+ 245us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.068918 (+ 210us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.069239 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":122,"threads_started":1}
I20260809 01:56:34.071573 29157 heartbeater.cc:347] Connected to a master server at 127.3.220.62:37457
I20260809 01:56:34.071846 29157 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:34.072438 29157 heartbeater.cc:511] Master 127.3.220.62:37457 requested a full tablet report, sending...
I20260809 01:56:34.073710 28668 ts_manager.cc:194] Registered new tserver with Master: 1a7ca721bb2748e7a9d7cdd77bc0222e (127.3.220.2:40745)
I20260809 01:56:34.074302 28668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:48881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:56:34.084821 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:37457
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:34.198581 29164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:34.198892 29164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:34.198972 29164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:34.202956 29164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:34.203056 29164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:34.203146 29164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:56:34.208500 29164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:34.209590 29164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:34.210625 29164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:34.220649 29174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:34.220680 29172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:34.220674 29176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:34.221071 29164 server_base.cc:1034] running on GCE node
I20260809 01:56:34.221426 29164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:34.222083 29164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:34.223248 29164 hybrid_clock.cc:648] HybridClock initialized: now 1786240594223210 us; error 49 us; skew 500 ppm
Aug 09 01:56:34 dist-test-slave-gkw7 krb5kdc[28610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:34.225996 29164 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:56:34.226801 29164 webserver.cc:460] Webserver started at http://127.3.220.3:40027/ using document root <none> and password file <none>
I20260809 01:56:34.227321 29164 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:34.227380 29164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:34.227540 29164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:34.229477 29164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "891c83324d3547bba909f5c18426177d"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "9b04241408617a06e4bf8de335e5582a"
server_key_iv: "19725f6f09c3d89781ec5d2c810861d6"
server_key_version: "encryptionkey@0"
I20260809 01:56:34.230069 29164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "891c83324d3547bba909f5c18426177d"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "9b04241408617a06e4bf8de335e5582a"
server_key_iv: "19725f6f09c3d89781ec5d2c810861d6"
server_key_version: "encryptionkey@0"
I20260809 01:56:34.233771 29164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:56:34.236187 29186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:34.237437 29164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:56:34.237633 29164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "891c83324d3547bba909f5c18426177d"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "9b04241408617a06e4bf8de335e5582a"
server_key_iv: "19725f6f09c3d89781ec5d2c810861d6"
server_key_version: "encryptionkey@0"
I20260809 01:56:34.237756 29164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:34.247661 29164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:34.250486 29164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:34.250686 29164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:34.251323 29164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:34.252120 29164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:34.252193 29164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:34.252314 29164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:34.252364 29164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:34.261484 29164 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:45755
I20260809 01:56:34.261556 29359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:45755 every 8 connection(s)
I20260809 01:56:34.262496 29164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:56:34.271363 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 29164
I20260809 01:56:34.271526 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:56:34.271848 3952 external_mini_cluster.cc:1428] Setting key b12e0e3e224b502cce95a7c91fcf7200
Aug 09 01:56:34 dist-test-slave-gkw7 krb5kdc[28610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:34.276885 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.264347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:42729 (local address 127.3.220.62:37457)
0809 01:56:34.264493 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.264497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.265314 (+ 817us) server_negotiation.cc:407] Connection header received
0809 01:56:34.266791 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.266795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.266851 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.266972 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:34.268228 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.268782 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.270109 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.270352 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.270477 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.273371 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:34.273374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:34.273377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:34.273403 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:34.275039 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:34.275523 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:34.275526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:34.275527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:34.275603 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:34.275914 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:34.275922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:34.275923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:34.276098 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:34.276174 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.276649 (+ 475us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.276752 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 01:56:34.277788 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.264707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:37457 (local address 127.3.220.3:42729)
0809 01:56:34.265156 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.265199 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.266574 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.267152 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.267160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.267586 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:34.268071 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.268085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.268944 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.268954 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.269973 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.269983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.270520 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.270524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.270700 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.271264 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:34.271287 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:34.273180 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:34.275149 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:34.275154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:34.275166 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:34.275430 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:34.275685 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:34.275687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:34.275688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:34.275828 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:34.276190 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:34.276195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:34.276438 (+ 243us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.276598 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.276855 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260809 01:56:34.278973 29360 heartbeater.cc:347] Connected to a master server at 127.3.220.62:37457
I20260809 01:56:34.279227 29360 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:34.279724 29360 heartbeater.cc:511] Master 127.3.220.62:37457 requested a full tablet report, sending...
I20260809 01:56:34.280774 28668 ts_manager.cc:194] Registered new tserver with Master: 891c83324d3547bba909f5c18426177d (127.3.220.3:45755)
I20260809 01:56:34.281312 28668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:42729
I20260809 01:56:34.286059 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:56:34.295493 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.288059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.3.220.62:37457)
0809 01:56:34.288203 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.288208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.288409 (+ 201us) server_negotiation.cc:407] Connection header received
0809 01:56:34.288507 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.288509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.288548 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.288614 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:34.289354 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.289958 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.290789 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.291005 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.291082 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.292386 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:34.292390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:34.292392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:34.292414 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:34.293951 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:34.294331 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:34.294334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:34.294335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:34.294411 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:34.294646 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:34.294648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:34.294649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:34.294866 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:34.294969 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.295177 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.295274 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:56:34.298439 28668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:56:34.301465 28668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:56:34.316311 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.310888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:36441 (local address 127.0.0.1:47230)
0809 01:56:34.311397 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.311410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.311486 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.312132 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.312135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.312155 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:34.312443 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.312447 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.313524 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.313528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.314708 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.314721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.315926 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.315932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.316027 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.316047 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.316133 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.316188 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":55,"thread_start_us":96,"threads_started":1}
I20260809 01:56:34.316313 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.310494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:40745 (local address 127.0.0.1:44190)
0809 01:56:34.310938 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.310988 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.311117 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.312157 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.312160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.312181 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:34.312315 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.312329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.313655 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.313658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.314762 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.314772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.315699 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.315702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.315827 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.315870 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.315978 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.316125 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":129,"thread_start_us":103,"threads_started":1}
I20260809 01:56:34.316431 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.311302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.3.220.1:36441)
0809 01:56:34.311657 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.311678 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.311697 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:56:34.311775 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.311782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.311970 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.312153 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:34.312557 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.313376 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.314861 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.315675 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.315795 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.315954 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.316189 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.316283 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":96,"threads_started":1}
I20260809 01:56:34.317058 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.310865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.3.220.2:40745)
0809 01:56:34.311675 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.311684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.311703 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:56:34.311765 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.311768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.311993 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.312259 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:34.312429 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.313498 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.314898 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.315551 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.315697 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.315847 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.316772 (+ 925us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.316893 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":130,"threads_started":1}
I20260809 01:56:34.317883 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.310339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.3.220.3:45755)
0809 01:56:34.311346 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.311359 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.311387 (+ 28us) server_negotiation.cc:407] Connection header received
0809 01:56:34.311474 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.311480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.311696 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.311866 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:34.312190 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.313430 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.315857 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.316886 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.316989 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.317566 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.317622 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.317733 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":143,"threads_started":1}
I20260809 01:56:34.318185 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.310112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:45755 (local address 127.0.0.1:59830)
0809 01:56:34.311263 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.311279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.311356 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.311876 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.311882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.311906 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:34.312049 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.312061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.313609 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.313615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.315718 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.315731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.317256 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.317263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.317376 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.317403 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.317473 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.318062 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":907,"thread_start_us":122,"threads_started":1}
I20260809 01:56:34.319667 28859 tablet_service.cc:1467] Processing CreateTablet for tablet f4338d5654ea45afbfa042ca3182488f (DEFAULT_TABLE table=test-table [id=52ba396a9b084b0a9e5bbd8482b5df93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:34.320060 29050 tablet_service.cc:1467] Processing CreateTablet for tablet f4338d5654ea45afbfa042ca3182488f (DEFAULT_TABLE table=test-table [id=52ba396a9b084b0a9e5bbd8482b5df93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:34.320799 28859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4338d5654ea45afbfa042ca3182488f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:34.321113 29050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4338d5654ea45afbfa042ca3182488f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:34.320880 29256 tablet_service.cc:1467] Processing CreateTablet for tablet f4338d5654ea45afbfa042ca3182488f (DEFAULT_TABLE table=test-table [id=52ba396a9b084b0a9e5bbd8482b5df93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:34.321882 29256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4338d5654ea45afbfa042ca3182488f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:34.327330 29389 tablet_bootstrap.cc:492] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9: Bootstrap starting.
I20260809 01:56:34.328987 29391 tablet_bootstrap.cc:492] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: Bootstrap starting.
I20260809 01:56:34.329780 29389 tablet_bootstrap.cc:654] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:34.329887 29393 tablet_bootstrap.cc:492] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Bootstrap starting.
I20260809 01:56:34.330582 29389 log.cc:824] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:34.331166 29391 tablet_bootstrap.cc:654] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:34.332056 29393 tablet_bootstrap.cc:654] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:34.332527 29391 log.cc:824] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:34.332615 29389 tablet_bootstrap.cc:492] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9: No bootstrap required, opened a new log
I20260809 01:56:34.332811 29389 ts_tablet_manager.cc:1397] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:56:34.333240 29393 log.cc:824] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:34.334682 29391 tablet_bootstrap.cc:492] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: No bootstrap required, opened a new log
I20260809 01:56:34.334858 29391 ts_tablet_manager.cc:1397] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:56:34.334864 29393 tablet_bootstrap.cc:492] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: No bootstrap required, opened a new log
I20260809 01:56:34.334995 29393 ts_tablet_manager.cc:1397] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 01:56:34.337242 29389 raft_consensus.cc:348] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.337563 29389 raft_consensus.cc:374] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:34.337628 29389 raft_consensus.cc:729] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f417a7d2690d43ad85bb859c1b9d37c9, State: Initialized, Role: FOLLOWER
I20260809 01:56:34.337688 29393 raft_consensus.cc:348] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.338073 29393 raft_consensus.cc:374] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:34.338084 29389 consensus_queue.cc:260] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.338158 29393 raft_consensus.cc:729] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891c83324d3547bba909f5c18426177d, State: Initialized, Role: FOLLOWER
I20260809 01:56:34.338704 29393 consensus_queue.cc:260] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.338833 28953 heartbeater.cc:503] Master 127.3.220.62:37457 was elected leader, sending a full tablet report...
I20260809 01:56:34.339141 29389 ts_tablet_manager.cc:1428] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 01:56:34.339622 29393 ts_tablet_manager.cc:1428] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260809 01:56:34.339583 29391 raft_consensus.cc:348] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.339895 29391 raft_consensus.cc:374] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:34.339948 29391 raft_consensus.cc:729] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7ca721bb2748e7a9d7cdd77bc0222e, State: Initialized, Role: FOLLOWER
I20260809 01:56:34.340008 29360 heartbeater.cc:503] Master 127.3.220.62:37457 was elected leader, sending a full tablet report...
I20260809 01:56:34.340379 29391 consensus_queue.cc:260] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.341009 29157 heartbeater.cc:503] Master 127.3.220.62:37457 was elected leader, sending a full tablet report...
I20260809 01:56:34.341280 29391 ts_tablet_manager.cc:1428] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:56:34.464772 29403 raft_consensus.cc:482] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:34.465039 29403 raft_consensus.cc:504] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.466390 29403 leader_election.cc:288] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 891c83324d3547bba909f5c18426177d (127.3.220.3:45755), 1a7ca721bb2748e7a9d7cdd77bc0222e (127.3.220.2:40745)
I20260809 01:56:34.470467 29408 raft_consensus.cc:482] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:34.470661 29408 raft_consensus.cc:504] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.470789 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.466805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:56151 (local address 127.3.220.2:40745)
0809 01:56:34.466949 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.466953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.467236 (+ 283us) server_negotiation.cc:407] Connection header received
0809 01:56:34.467481 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.467487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.467538 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.467629 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:34.467944 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.468456 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.469737 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.470153 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.470284 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.470348 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.470557 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.470650 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:56:34.470909 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.466783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:50423 (local address 127.3.220.3:45755)
0809 01:56:34.466941 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.466946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.467388 (+ 442us) server_negotiation.cc:407] Connection header received
0809 01:56:34.467542 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.467545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.467598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.467687 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:34.467968 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.468459 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.469727 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.470202 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.470315 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.470373 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.470707 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.470777 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 01:56:34.471169 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.466633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:40745 (local address 127.3.220.1:56151)
0809 01:56:34.467121 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.467134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.467286 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.467642 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.467645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.467662 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:34.467809 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.467816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.468576 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.468580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.469599 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.469609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.470294 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.470298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.470396 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.470421 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.470507 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.470698 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":40,"thread_start_us":220,"threads_started":1}
I20260809 01:56:34.471243 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.466633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:45755 (local address 127.3.220.1:50423)
0809 01:56:34.467270 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.467283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.467424 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.467700 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.467703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.467720 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:34.467852 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.467858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.468578 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.468582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.469600 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.469610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.470428 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.470430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.470561 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.470585 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.471035 (+ 450us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.471109 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":140,"thread_start_us":137,"threads_started":1}
I20260809 01:56:34.471863 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4338d5654ea45afbfa042ca3182488f" candidate_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891c83324d3547bba909f5c18426177d" is_pre_election: true
I20260809 01:56:34.471959 29079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4338d5654ea45afbfa042ca3182488f" candidate_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" is_pre_election: true
I20260809 01:56:34.472065 29293 raft_consensus.cc:2454] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f417a7d2690d43ad85bb859c1b9d37c9 in term 0.
I20260809 01:56:34.472313 29079 raft_consensus.cc:2454] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f417a7d2690d43ad85bb859c1b9d37c9 in term 0.
I20260809 01:56:34.472558 29408 leader_election.cc:288] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f417a7d2690d43ad85bb859c1b9d37c9 (127.3.220.1:36441), 1a7ca721bb2748e7a9d7cdd77bc0222e (127.3.220.2:40745)
I20260809 01:56:34.473099 28785 leader_election.cc:302] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a7ca721bb2748e7a9d7cdd77bc0222e, f417a7d2690d43ad85bb859c1b9d37c9; no voters:
I20260809 01:56:34.473590 29403 raft_consensus.cc:2781] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:56:34.473692 29403 raft_consensus.cc:482] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:56:34.473742 29403 raft_consensus.cc:3037] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:34.475149 29403 raft_consensus.cc:504] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.475560 29403 leader_election.cc:288] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 891c83324d3547bba909f5c18426177d (127.3.220.3:45755), 1a7ca721bb2748e7a9d7cdd77bc0222e (127.3.220.2:40745)
I20260809 01:56:34.475852 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4338d5654ea45afbfa042ca3182488f" candidate_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891c83324d3547bba909f5c18426177d"
I20260809 01:56:34.475978 29293 raft_consensus.cc:3037] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:34.476006 29079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4338d5654ea45afbfa042ca3182488f" candidate_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e"
I20260809 01:56:34.476135 29079 raft_consensus.cc:3037] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:34.476955 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.472871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:44771 (local address 127.3.220.1:36441)
0809 01:56:34.472987 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.472992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.473354 (+ 362us) server_negotiation.cc:407] Connection header received
0809 01:56:34.473571 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.473574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.473628 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.473714 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:34.474095 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.474574 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.475821 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.476377 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.476436 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.476476 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.476785 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.476858 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 01:56:34.476974 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.472802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:36441 (local address 127.3.220.3:44771)
0809 01:56:34.473259 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.473273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.473418 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.473846 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.473849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.473868 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:34.473992 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.474001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.474699 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.474703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.475684 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.475696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.476580 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.476582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.476660 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.476676 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.476763 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.476832 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":121,"threads_started":1}
I20260809 01:56:34.477335 29293 raft_consensus.cc:2454] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f417a7d2690d43ad85bb859c1b9d37c9 in term 1.
I20260809 01:56:34.477454 29079 raft_consensus.cc:2454] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f417a7d2690d43ad85bb859c1b9d37c9 in term 1.
I20260809 01:56:34.477805 28790 leader_election.cc:302] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 891c83324d3547bba909f5c18426177d, f417a7d2690d43ad85bb859c1b9d37c9; no voters:
I20260809 01:56:34.478147 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.472487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:49553 (local address 127.3.220.2:40745)
0809 01:56:34.472606 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.472610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.472624 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:56:34.472668 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.472673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.472728 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.472797 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:34.473183 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.473901 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.475377 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.476038 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.476165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.476220 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.477999 (+ 1779us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.478064 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 01:56:34.478174 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.472304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:40745 (local address 127.3.220.3:49553)
0809 01:56:34.472421 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.472436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:34.472515 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:34.472925 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:34.472931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:34.472953 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:34.473055 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.473064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.474038 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.474042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.475193 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:34.475204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.477713 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.477717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:34.477841 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.477894 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 01:56:34.477981 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 01:56:34.478062 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 01:56:34.478223 28884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4338d5654ea45afbfa042ca3182488f" candidate_uuid: "891c83324d3547bba909f5c18426177d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" is_pre_election: true
I20260809 01:56:34.478502 28884 raft_consensus.cc:2379] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 891c83324d3547bba909f5c18426177d in current term 1: Already voted for candidate f417a7d2690d43ad85bb859c1b9d37c9 in this term.
I20260809 01:56:34.478700 29403 raft_consensus.cc:2781] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:34.478704 29079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4338d5654ea45afbfa042ca3182488f" candidate_uuid: "891c83324d3547bba909f5c18426177d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" is_pre_election: true
I20260809 01:56:34.478876 29079 raft_consensus.cc:2379] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 891c83324d3547bba909f5c18426177d in current term 1: Already voted for candidate f417a7d2690d43ad85bb859c1b9d37c9 in this term.
I20260809 01:56:34.478966 29403 raft_consensus.cc:686] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 1 LEADER]: Becoming Leader. State: Replica: f417a7d2690d43ad85bb859c1b9d37c9, State: Running, Role: LEADER
I20260809 01:56:34.479328 29403 consensus_queue.cc:237] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } }
I20260809 01:56:34.479377 29187 leader_election.cc:302] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 891c83324d3547bba909f5c18426177d; no voters: 1a7ca721bb2748e7a9d7cdd77bc0222e, f417a7d2690d43ad85bb859c1b9d37c9
I20260809 01:56:34.479650 29408 raft_consensus.cc:2726] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 01:56:34.482609 28668 catalog_manager.cc:5495] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f417a7d2690d43ad85bb859c1b9d37c9 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f417a7d2690d43ad85bb859c1b9d37c9" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 } health_report { overall_health: UNKNOWN } } }
I20260809 01:56:34.506042 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.502037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.3.220.1:36441)
0809 01:56:34.502175 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.502179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.502196 (+ 17us) server_negotiation.cc:407] Connection header received
0809 01:56:34.502329 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.502332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.502396 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.502501 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:56:34.502731 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.503547 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.504412 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.504761 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.504863 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.505134 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:56:34.505676 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:56:34.505796 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.505856 (+ 60us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.505911 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 01:56:34.512293 29079 raft_consensus.cc:1261] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 1 FOLLOWER]: Refusing update from remote peer f417a7d2690d43ad85bb859c1b9d37c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:34.512343 29293 raft_consensus.cc:1261] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 1 FOLLOWER]: Refusing update from remote peer f417a7d2690d43ad85bb859c1b9d37c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:34.513047 29414 consensus_queue.cc:1035] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a7ca721bb2748e7a9d7cdd77bc0222e" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:34.513239 29403 consensus_queue.cc:1035] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "891c83324d3547bba909f5c18426177d" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 01:56:34.513504 29363 tablet.cc:2369] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:56:34.531275 28668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:34.531432 28668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:34.533934 28668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87b4ae6528194e27b3f3c8429ae00f37: Sending DeleteTablet for 3 replicas of tablet f4338d5654ea45afbfa042ca3182488f
I20260809 01:56:34.534790 29050 tablet_service.cc:1514] Processing DeleteTablet for tablet f4338d5654ea45afbfa042ca3182488f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:34 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:44190
I20260809 01:56:34.534777 28859 tablet_service.cc:1514] Processing DeleteTablet for tablet f4338d5654ea45afbfa042ca3182488f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:34 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:47230
I20260809 01:56:34.534816 29256 tablet_service.cc:1514] Processing DeleteTablet for tablet f4338d5654ea45afbfa042ca3182488f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:34 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:59830
I20260809 01:56:34.535280 29431 tablet_replica.cc:330] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: stopping tablet replica
I20260809 01:56:34.535281 29433 tablet_replica.cc:330] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9: stopping tablet replica
I20260809 01:56:34.535598 29433 raft_consensus.cc:2229] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:56:34.535598 29431 raft_consensus.cc:2229] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:34.535915 29431 raft_consensus.cc:2258] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:34.535964 29433 raft_consensus.cc:2258] T f4338d5654ea45afbfa042ca3182488f P f417a7d2690d43ad85bb859c1b9d37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:34.536492 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 28758
I20260809 01:56:34.537592 29431 ts_tablet_manager.cc:1905] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:34.541909 29431 ts_tablet_manager.cc:1918] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:34.542016 29431 log.cc:1196] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/f4338d5654ea45afbfa042ca3182488f
I20260809 01:56:34.542304 29431 ts_tablet_manager.cc:1939] T f4338d5654ea45afbfa042ca3182488f P 1a7ca721bb2748e7a9d7cdd77bc0222e: Deleting consensus metadata
I20260809 01:56:34.543587 28648 catalog_manager.cc:4841] TS 1a7ca721bb2748e7a9d7cdd77bc0222e (127.3.220.2:40745): tablet f4338d5654ea45afbfa042ca3182488f (table test-table [id=52ba396a9b084b0a9e5bbd8482b5df93]) successfully deleted
I20260809 01:56:34.544631 29434 tablet_replica.cc:330] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: stopping tablet replica
I20260809 01:56:34.544961 29434 raft_consensus.cc:2229] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:34.545281 29434 raft_consensus.cc:2258] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:34.547503 29434 ts_tablet_manager.cc:1905] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:34.551482 29434 ts_tablet_manager.cc:1918] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:34.551597 29434 log.cc:1196] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/f4338d5654ea45afbfa042ca3182488f
I20260809 01:56:34.551952 29434 ts_tablet_manager.cc:1939] T f4338d5654ea45afbfa042ca3182488f P 891c83324d3547bba909f5c18426177d: Deleting consensus metadata
I20260809 01:56:34.553442 28650 catalog_manager.cc:4841] TS 891c83324d3547bba909f5c18426177d (127.3.220.3:45755): tablet f4338d5654ea45afbfa042ca3182488f (table test-table [id=52ba396a9b084b0a9e5bbd8482b5df93]) successfully deleted
W20260809 01:56:34.554271 28650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:36441 (error 108)
I20260809 01:56:34.554555 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 28960
I20260809 01:56:34.557513 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.557069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:36441 (local address 127.0.0.1:47246)
0809 01:56:34.557259 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:34.557389 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 01:56:34.558099 28650 catalog_manager.cc:4584] TS f417a7d2690d43ad85bb859c1b9d37c9 (127.3.220.1:36441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4338d5654ea45afbfa042ca3182488f: Network error: Client connection negotiation failed: client connection to 127.3.220.1:36441: connect: Connection refused (error 111)
I20260809 01:56:34.566200 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 29164
I20260809 01:56:34.578270 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 28627
2026-08-09T01:56:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2661 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T01:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:34Z Disabled control of system clock
I20260809 01:56:34.615531 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41413
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 01:56:34.734755 29441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:34.735011 29441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:34.735049 29441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:34.738897 29441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:34.738946 29441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:34.738962 29441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:34.738978 29441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:34.738993 29441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:34.743784 29441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:41413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:34.744760 29441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:34.746097 29441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:34.754730 29453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:34.754736 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:34.754736 29451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:34.755066 29441 server_base.cc:1034] running on GCE node
I20260809 01:56:34.755587 29441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:34.756647 29441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:34.757800 29441 hybrid_clock.cc:648] HybridClock initialized: now 1786240594757789 us; error 36 us; skew 500 ppm
I20260809 01:56:34.759660 29441 webserver.cc:460] Webserver started at http://127.3.220.62:39569/ using document root <none> and password file <none>
I20260809 01:56:34.760263 29441 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:34.760329 29441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:34.760577 29441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:34.762733 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "3e25e84136664a3cb7374fa9f9116db8"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "07947fac45fc2b9174e2653ea0929dd0"
server_key_iv: "d86b1987d7f6636543fa2d34722a2c95"
server_key_version: "encryptionkey@0"
I20260809 01:56:34.763309 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "3e25e84136664a3cb7374fa9f9116db8"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "07947fac45fc2b9174e2653ea0929dd0"
server_key_iv: "d86b1987d7f6636543fa2d34722a2c95"
server_key_version: "encryptionkey@0"
I20260809 01:56:34.766996 29441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260809 01:56:34.769404 29461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:34.770624 29441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:56:34.770767 29441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "3e25e84136664a3cb7374fa9f9116db8"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "07947fac45fc2b9174e2653ea0929dd0"
server_key_iv: "d86b1987d7f6636543fa2d34722a2c95"
server_key_version: "encryptionkey@0"
I20260809 01:56:34.770887 29441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:34.783385 29441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:34.784042 29441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:34.784237 29441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:34.791652 29441 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:41413
I20260809 01:56:34.791651 29539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:41413 every 8 connection(s)
I20260809 01:56:34.792605 29441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:34.795733 29541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:34.801692 29541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8: Bootstrap starting.
I20260809 01:56:34.802057 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 29441
I20260809 01:56:34.802194 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:34.802467 3952 external_mini_cluster.cc:1428] Setting key 2dbe55866fd601bb5ec84f148ab8b7fa
I20260809 01:56:34.804394 29541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:34.805197 29541 log.cc:824] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:34.807163 29541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8: No bootstrap required, opened a new log
I20260809 01:56:34.810640 29541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e84136664a3cb7374fa9f9116db8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41413 } }
I20260809 01:56:34.810882 29541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:34.810938 29541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e25e84136664a3cb7374fa9f9116db8, State: Initialized, Role: FOLLOWER
I20260809 01:56:34.811362 29541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e84136664a3cb7374fa9f9116db8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41413 } }
I20260809 01:56:34.811540 29541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:34.811637 29541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:34.811764 29541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:34.812095 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:34.803808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.3.220.62:41413)
0809 01:56:34.804541 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:34.804564 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:34.804612 (+ 48us) server_negotiation.cc:407] Connection header received
0809 01:56:34.805251 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:34.805268 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:34.805819 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:34.806152 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:34.806578 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.807606 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.808515 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:34.809002 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:34.809177 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:34.810091 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:34.810103 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:34.810117 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:34.810527 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:34.810585 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:34.810608 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:34.810721 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:34.810847 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 01:56:34.811130 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260809 01:56:34.812768 29541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e84136664a3cb7374fa9f9116db8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41413 } }
I20260809 01:56:34.813146 29541 leader_election.cc:302] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e25e84136664a3cb7374fa9f9116db8; no voters:
I20260809 01:56:34.813511 29541 leader_election.cc:288] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:34.813637 29550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:34.813836 29550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e25e84136664a3cb7374fa9f9116db8, State: Running, Role: LEADER
I20260809 01:56:34.814169 29550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e84136664a3cb7374fa9f9116db8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41413 } }
I20260809 01:56:34.814880 29541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:34.815248 29550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e25e84136664a3cb7374fa9f9116db8. Latest consensus state: current_term: 1 leader_uuid: "3e25e84136664a3cb7374fa9f9116db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e84136664a3cb7374fa9f9116db8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41413 } } }
I20260809 01:56:34.815372 29550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:34.815758 29556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:34.815683 29551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e25e84136664a3cb7374fa9f9116db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e25e84136664a3cb7374fa9f9116db8" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41413 } } }
I20260809 01:56:34.815845 29551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:34.818647 29556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:34.824100 29556 catalog_manager.cc:1347] Generated new cluster ID: 3ddb458a11b2406985ab9335deaa25cd
I20260809 01:56:34.824172 29556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:34.844017 29556 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:34.845345 29556 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:34.850930 29556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8: Generated new TSK 0
I20260809 01:56:34.851617 29556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:34.855799 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41413
--builtin_ntp_servers=127.3.220.20:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 01:56:34.970893 29578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:34.971132 29578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:34.971169 29578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:34.975255 29578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:34.975311 29578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:34.975404 29578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:56:34.980111 29578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:34.981227 29578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:34.982693 29578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:34.992763 29587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:34.992833 29588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:34.992842 29578 server_base.cc:1034] running on GCE node
W20260809 01:56:34.992789 29591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:34.993633 29578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:34.994287 29578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:34.995462 29578 hybrid_clock.cc:648] HybridClock initialized: now 1786240594995445 us; error 29 us; skew 500 ppm
I20260809 01:56:34.997277 29578 webserver.cc:460] Webserver started at http://127.3.220.1:38133/ using document root <none> and password file <none>
I20260809 01:56:34.997939 29578 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:34.998003 29578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:34.998257 29578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:35.000315 29578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "eea8174356614449aa30ee22d125389f"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "58d1cddc22bdd73a4bc915429bf96009"
server_key_iv: "5c589df2c096453fceece5bafd48a843"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.000907 29578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "eea8174356614449aa30ee22d125389f"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "58d1cddc22bdd73a4bc915429bf96009"
server_key_iv: "5c589df2c096453fceece5bafd48a843"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.004628 29578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 01:56:35.007097 29605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.008297 29578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:56:35.008432 29578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "eea8174356614449aa30ee22d125389f"
format_stamp: "Formatted at 2026-08-09 01:56:34 on dist-test-slave-gkw7"
server_key: "58d1cddc22bdd73a4bc915429bf96009"
server_key_iv: "5c589df2c096453fceece5bafd48a843"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.008538 29578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:35.033666 29578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:35.034427 29578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:35.034639 29578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:35.035269 29578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:35.036249 29578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:35.036296 29578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.036378 29578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:35.036410 29578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.045874 29578 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:35901
I20260809 01:56:35.045913 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:35901 every 8 connection(s)
I20260809 01:56:35.046820 29578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:56:35.054064 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 29578
I20260809 01:56:35.054219 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:56:35.054536 3952 external_mini_cluster.cc:1428] Setting key 72fbe7f60897fd1061e33f68b1d34a23
I20260809 01:56:35.056226 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.048739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:50233 (local address 127.3.220.62:41413)
0809 01:56:35.048880 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.048884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.050025 (+ 1141us) server_negotiation.cc:407] Connection header received
0809 01:56:35.051393 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.051396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.051444 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.051521 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:35.052364 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.052900 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.054077 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.054410 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.054535 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.055648 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:35.055652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:35.055654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:35.055668 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:35.055692 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:35.055702 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:35.055795 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.055998 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.056063 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 01:56:35.056807 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41413
--builtin_ntp_servers=127.3.220.20:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 01:56:35.057323 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.049144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41413 (local address 127.3.220.1:50233)
0809 01:56:35.049860 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.049902 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:35.051186 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:35.051658 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:35.051665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:35.051748 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:35.052230 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.052242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.053014 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.053016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.053927 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.053937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.054580 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.054584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.054755 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.055369 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:35.055372 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:35.055428 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:35.055494 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:35.055795 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:35.055834 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:35.055841 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 01:56:35.055975 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 01:56:35.056281 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":117,"threads_started":1}
I20260809 01:56:35.058736 29802 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41413
I20260809 01:56:35.059018 29802 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:35.059549 29802 heartbeater.cc:511] Master 127.3.220.62:41413 requested a full tablet report, sending...
I20260809 01:56:35.061094 29485 ts_manager.cc:194] Registered new tserver with Master: eea8174356614449aa30ee22d125389f (127.3.220.1:35901)
I20260809 01:56:35.062345 29485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.1:50233
W20260809 01:56:35.174209 29808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:35.174556 29808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:35.174600 29808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:35.178643 29808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:35.178699 29808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:35.178755 29808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:56:35.183466 29808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:35.184578 29808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:35.185972 29808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:35.195561 29820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:35.195531 29823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:35.195554 29819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:35.195936 29808 server_base.cc:1034] running on GCE node
I20260809 01:56:35.196367 29808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:35.197010 29808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:35.198204 29808 hybrid_clock.cc:648] HybridClock initialized: now 1786240595198185 us; error 38 us; skew 500 ppm
I20260809 01:56:35.200078 29808 webserver.cc:460] Webserver started at http://127.3.220.2:44399/ using document root <none> and password file <none>
I20260809 01:56:35.200681 29808 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:35.200780 29808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:35.201005 29808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:35.203135 29808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "fd445070c97e419e97a68d8b5660f316"
format_stamp: "Formatted at 2026-08-09 01:56:35 on dist-test-slave-gkw7"
server_key: "d17407cc9cc6eca607fa21ba594e15a3"
server_key_iv: "45613878f9d4935d89297bb0932a1524"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.203720 29808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "fd445070c97e419e97a68d8b5660f316"
format_stamp: "Formatted at 2026-08-09 01:56:35 on dist-test-slave-gkw7"
server_key: "d17407cc9cc6eca607fa21ba594e15a3"
server_key_iv: "45613878f9d4935d89297bb0932a1524"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.207595 29808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:56:35.210148 29836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.211452 29808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:35.211592 29808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "fd445070c97e419e97a68d8b5660f316"
format_stamp: "Formatted at 2026-08-09 01:56:35 on dist-test-slave-gkw7"
server_key: "d17407cc9cc6eca607fa21ba594e15a3"
server_key_iv: "45613878f9d4935d89297bb0932a1524"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.211707 29808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:35.232982 29808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:35.233762 29808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:35.233987 29808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:35.234671 29808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:35.235646 29808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:35.235721 29808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.235791 29808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:35.235847 29808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.245260 29808 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:38141
I20260809 01:56:35.245280 30013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:38141 every 8 connection(s)
I20260809 01:56:35.246258 29808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:56:35.255143 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 29808
I20260809 01:56:35.255247 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:56:35.255561 3952 external_mini_cluster.cc:1428] Setting key fb5e2de6b6ecc68c2dd00b9073643f89
I20260809 01:56:35.255954 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.248215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:45735 (local address 127.3.220.62:41413)
0809 01:56:35.248373 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.248377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.249508 (+ 1131us) server_negotiation.cc:407] Connection header received
0809 01:56:35.250964 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.250967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.251028 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.251111 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:35.252017 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.252569 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.253788 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.254037 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.254156 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.255428 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:35.255432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:35.255434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:35.255446 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:35.255466 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:35.255474 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:35.255556 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.255738 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.255803 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:56:35.256805 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.248600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41413 (local address 127.3.220.2:45735)
0809 01:56:35.249349 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.249390 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:35.250735 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:35.251295 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:35.251302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:35.251383 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:35.251860 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.251872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.252716 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.252720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.253653 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.253662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.254201 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.254205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.254368 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.255091 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:35.255093 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:35.255141 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:35.255231 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:35.255559 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:35.255571 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:35.255580 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 01:56:35.255718 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 01:56:35.256004 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":123,"threads_started":1}
I20260809 01:56:35.257933 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41413
--builtin_ntp_servers=127.3.220.20:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 01:56:35.258245 30014 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41413
I20260809 01:56:35.258477 30014 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:35.259102 30014 heartbeater.cc:511] Master 127.3.220.62:41413 requested a full tablet report, sending...
I20260809 01:56:35.260217 29485 ts_manager.cc:194] Registered new tserver with Master: fd445070c97e419e97a68d8b5660f316 (127.3.220.2:38141)
I20260809 01:56:35.260814 29485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.2:45735
W20260809 01:56:35.377908 30021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:35.378218 30021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:35.378289 30021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:35.382297 30021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:35.382386 30021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:35.382475 30021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:56:35.387244 30021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:41413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:35.388422 30021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:35.389947 30021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:56:35.400260 30021 server_base.cc:1034] running on GCE node
W20260809 01:56:35.400209 30033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:35.400208 30038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:35.400219 30035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:35.400986 30021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:35.401633 30021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:35.402794 30021 hybrid_clock.cc:648] HybridClock initialized: now 1786240595402759 us; error 48 us; skew 500 ppm
I20260809 01:56:35.404647 30021 webserver.cc:460] Webserver started at http://127.3.220.3:34927/ using document root <none> and password file <none>
I20260809 01:56:35.405261 30021 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:35.405329 30021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:35.405684 30021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:35.407686 30021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "2c02926848e14920af388c429e94af63"
format_stamp: "Formatted at 2026-08-09 01:56:35 on dist-test-slave-gkw7"
server_key: "01a035bd8968051722726bc1afbca9ff"
server_key_iv: "bd935ec8de8dd0d8f35c989b50842c59"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.408226 30021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "2c02926848e14920af388c429e94af63"
format_stamp: "Formatted at 2026-08-09 01:56:35 on dist-test-slave-gkw7"
server_key: "01a035bd8968051722726bc1afbca9ff"
server_key_iv: "bd935ec8de8dd0d8f35c989b50842c59"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.411908 30021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:56:35.414495 30046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.415807 30021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 01:56:35.415946 30021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "2c02926848e14920af388c429e94af63"
format_stamp: "Formatted at 2026-08-09 01:56:35 on dist-test-slave-gkw7"
server_key: "01a035bd8968051722726bc1afbca9ff"
server_key_iv: "bd935ec8de8dd0d8f35c989b50842c59"
server_key_version: "encryptionkey@0"
I20260809 01:56:35.416080 30021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:35.439432 30021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:35.440249 30021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:35.440488 30021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:35.441154 30021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:35.442281 30021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:35.442358 30021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.442430 30021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:35.442477 30021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:35.451854 30021 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:46867
I20260809 01:56:35.451879 30229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:46867 every 8 connection(s)
I20260809 01:56:35.452788 30021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:56:35.456425 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 30021
I20260809 01:56:35.456514 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:56:35.456805 3952 external_mini_cluster.cc:1428] Setting key 2b8a1f97a3422f3d085841eb859683d5
I20260809 01:56:35.462739 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.454751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:46393 (local address 127.3.220.62:41413)
0809 01:56:35.454905 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.454909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.456025 (+ 1116us) server_negotiation.cc:407] Connection header received
0809 01:56:35.457773 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.457776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.457830 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.457907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:35.458996 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.459621 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.460784 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.461011 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.461087 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.462327 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:35.462330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:35.462332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:35.462343 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:35.462361 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:35.462368 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:35.462427 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.462573 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.462624 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 01:56:35.463625 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.455146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41413 (local address 127.3.220.3:46393)
0809 01:56:35.455853 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.455895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:35.457536 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:35.458098 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:35.458105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:35.458196 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:35.458846 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.458866 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.459754 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.459757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.460676 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.460686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.461176 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.461179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.461338 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.462084 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:35.462085 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:35.462126 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:35.462192 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:35.462447 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:35.462455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:35.462460 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 01:56:35.462558 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 01:56:35.462826 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260809 01:56:35.464780 30230 heartbeater.cc:347] Connected to a master server at 127.3.220.62:41413
I20260809 01:56:35.465032 30230 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:35.465641 30230 heartbeater.cc:511] Master 127.3.220.62:41413 requested a full tablet report, sending...
I20260809 01:56:35.466665 29485 ts_manager.cc:194] Registered new tserver with Master: 2c02926848e14920af388c429e94af63 (127.3.220.3:46867)
I20260809 01:56:35.467201 29485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.3:46393
I20260809 01:56:35.470885 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:56:35.476351 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.472817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.3.220.62:41413)
0809 01:56:35.472948 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.472952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.473139 (+ 187us) server_negotiation.cc:407] Connection header received
0809 01:56:35.473229 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.473231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.473268 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.473342 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:35.473644 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.474223 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.475020 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.475279 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.475367 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.476036 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:35.476039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:35.476041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:35.476052 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:35.476070 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:35.476077 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:35.476137 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.476190 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.476236 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:56:35.479058 29485 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:56:35.482152 29485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:56:35.496866 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.491397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:35901 (local address 127.0.0.1:54256)
0809 01:56:35.492012 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.492025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:35.492111 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:35.492838 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:35.492841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:35.492862 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:35.492957 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.492963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.494060 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.494066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.495274 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.495285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.496268 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.496273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.496402 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.496446 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 01:56:35.496556 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 01:56:35.496687 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":85,"thread_start_us":95,"threads_started":1}
I20260809 01:56:35.497205 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.491960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.3.220.1:35901)
0809 01:56:35.492369 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.492381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.492404 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:56:35.492481 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.492486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.492672 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.492830 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:35.493088 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.493941 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.495421 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.496116 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.496230 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.496429 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.496949 (+ 520us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.497043 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":105,"threads_started":1}
I20260809 01:56:35.497474 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.490742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:46867 (local address 127.0.0.1:50890)
0809 01:56:35.491538 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.491554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:35.491691 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:35.492239 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:35.492244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:35.492265 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:35.492411 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.492419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.494172 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.494175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.495564 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.495575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.496553 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.496557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.496654 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.496676 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 01:56:35.496752 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 01:56:35.496836 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":58,"thread_start_us":107,"threads_started":1}
I20260809 01:56:35.497860 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.491946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.3.220.2:38141)
0809 01:56:35.492365 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.492378 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.492403 (+ 25us) server_negotiation.cc:407] Connection header received
0809 01:56:35.492481 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.492486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.492672 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.492837 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:35.493301 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.494387 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.495769 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.496541 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.496634 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.496806 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.497473 (+ 667us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.497682 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":122,"threads_started":1}
I20260809 01:56:35.498064 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.490622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38141 (local address 127.0.0.1:36698)
0809 01:56:35.491478 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.491526 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:35.491660 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:35.492993 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:35.492996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:35.493046 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:35.493147 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.493153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.494518 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.494521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.495647 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.495656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.496867 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.496871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.496959 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.497162 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 01:56:35.497239 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 01:56:35.497934 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260809 01:56:35.498085 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.490963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.3.220.3:46867)
0809 01:56:35.491418 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.491430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.491631 (+ 201us) server_negotiation.cc:407] Connection header received
0809 01:56:35.491851 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.491857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.492094 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.492252 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:35.492546 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.494052 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.495718 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.496332 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.496411 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.496587 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.496813 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.496927 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":112,"threads_started":1}
I20260809 01:56:35.500377 30115 tablet_service.cc:1467] Processing CreateTablet for tablet cb951d4e0f6648b2b701509a79e88f93 (DEFAULT_TABLE table=test-table [id=81ca0d671c894dbfb5d96a2dc79af42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:35.500422 29679 tablet_service.cc:1467] Processing CreateTablet for tablet cb951d4e0f6648b2b701509a79e88f93 (DEFAULT_TABLE table=test-table [id=81ca0d671c894dbfb5d96a2dc79af42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:35.500679 29910 tablet_service.cc:1467] Processing CreateTablet for tablet cb951d4e0f6648b2b701509a79e88f93 (DEFAULT_TABLE table=test-table [id=81ca0d671c894dbfb5d96a2dc79af42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:35.501504 29910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb951d4e0f6648b2b701509a79e88f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:35.501504 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb951d4e0f6648b2b701509a79e88f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:35.501504 29679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb951d4e0f6648b2b701509a79e88f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:35.507907 30256 tablet_bootstrap.cc:492] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: Bootstrap starting.
I20260809 01:56:35.509240 30257 tablet_bootstrap.cc:492] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f: Bootstrap starting.
I20260809 01:56:35.509356 30259 tablet_bootstrap.cc:492] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: Bootstrap starting.
I20260809 01:56:35.510298 30256 tablet_bootstrap.cc:654] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:35.511044 30256 log.cc:824] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:35.511439 30257 tablet_bootstrap.cc:654] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:35.511439 30259 tablet_bootstrap.cc:654] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:35.512274 30259 log.cc:824] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:35.512286 30257 log.cc:824] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:35.513870 30256 tablet_bootstrap.cc:492] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: No bootstrap required, opened a new log
I20260809 01:56:35.514091 30256 ts_tablet_manager.cc:1397] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260809 01:56:35.514308 30257 tablet_bootstrap.cc:492] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f: No bootstrap required, opened a new log
I20260809 01:56:35.514555 30257 ts_tablet_manager.cc:1397] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 01:56:35.514871 30259 tablet_bootstrap.cc:492] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: No bootstrap required, opened a new log
I20260809 01:56:35.515141 30259 ts_tablet_manager.cc:1397] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:56:35.518059 30256 raft_consensus.cc:348] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.518394 30256 raft_consensus.cc:374] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:35.518450 30256 raft_consensus.cc:729] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c02926848e14920af388c429e94af63, State: Initialized, Role: FOLLOWER
I20260809 01:56:35.518941 30256 consensus_queue.cc:260] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.518919 30259 raft_consensus.cc:348] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.518919 30257 raft_consensus.cc:348] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.519161 30257 raft_consensus.cc:374] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:35.519161 30259 raft_consensus.cc:374] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:35.519223 30259 raft_consensus.cc:729] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd445070c97e419e97a68d8b5660f316, State: Initialized, Role: FOLLOWER
I20260809 01:56:35.519223 30257 raft_consensus.cc:729] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea8174356614449aa30ee22d125389f, State: Initialized, Role: FOLLOWER
I20260809 01:56:35.519660 30259 consensus_queue.cc:260] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.519711 30257 consensus_queue.cc:260] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.520198 30256 ts_tablet_manager.cc:1428] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260809 01:56:35.520457 29802 heartbeater.cc:503] Master 127.3.220.62:41413 was elected leader, sending a full tablet report...
I20260809 01:56:35.520536 30230 heartbeater.cc:503] Master 127.3.220.62:41413 was elected leader, sending a full tablet report...
I20260809 01:56:35.520639 30259 ts_tablet_manager.cc:1428] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 01:56:35.521018 30014 heartbeater.cc:503] Master 127.3.220.62:41413 was elected leader, sending a full tablet report...
I20260809 01:56:35.522017 30257 ts_tablet_manager.cc:1428] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 01:56:35.547941 30265 raft_consensus.cc:482] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 01:56:35.548110 29803 tablet.cc:2369] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:56:35.548189 30265 raft_consensus.cc:504] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.549407 30265 leader_election.cc:288] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd445070c97e419e97a68d8b5660f316 (127.3.220.2:38141), eea8174356614449aa30ee22d125389f (127.3.220.1:35901)
I20260809 01:56:35.553433 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.550100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:37885 (local address 127.3.220.2:38141)
0809 01:56:35.550249 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.550253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.550266 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:56:35.550331 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.550334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.550395 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.550490 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:35.550782 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.551298 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.552486 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.552873 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.552999 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.553060 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.553241 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.553323 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 01:56:35.553433 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.549654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38141 (local address 127.3.220.3:37885)
0809 01:56:35.549882 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.549900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:35.549995 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:35.550509 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:35.550512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:35.550531 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:35.550656 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.550662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.551421 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.551426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.552355 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.552365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.553006 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.553009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.553112 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.553130 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 01:56:35.553225 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 01:56:35.553297 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260809 01:56:35.553565 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.549847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:44453 (local address 127.3.220.1:35901)
0809 01:56:35.549988 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.549993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.550291 (+ 298us) server_negotiation.cc:407] Connection header received
0809 01:56:35.550562 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.550565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.550610 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.550699 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:35.550983 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.551482 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.552538 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.552943 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.553073 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.553127 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.553327 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.553396 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 01:56:35.553622 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.549715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:35901 (local address 127.3.220.3:44453)
0809 01:56:35.550200 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.550217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:35.550415 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:35.550711 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:35.550714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:35.550731 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:35.550859 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.550864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.551598 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.551602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.552399 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:35.552408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.553087 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.553091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:35.553197 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.553215 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 01:56:35.553308 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 01:56:35.553386 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":165,"threads_started":1}
I20260809 01:56:35.554222 29941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb951d4e0f6648b2b701509a79e88f93" candidate_uuid: "2c02926848e14920af388c429e94af63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd445070c97e419e97a68d8b5660f316" is_pre_election: true
I20260809 01:56:35.554221 29707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb951d4e0f6648b2b701509a79e88f93" candidate_uuid: "2c02926848e14920af388c429e94af63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea8174356614449aa30ee22d125389f" is_pre_election: true
I20260809 01:56:35.554549 29941 raft_consensus.cc:2454] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c02926848e14920af388c429e94af63 in term 0.
I20260809 01:56:35.554549 29707 raft_consensus.cc:2454] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c02926848e14920af388c429e94af63 in term 0.
I20260809 01:56:35.555083 30048 leader_election.cc:302] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c02926848e14920af388c429e94af63, eea8174356614449aa30ee22d125389f; no voters:
I20260809 01:56:35.555346 30265 raft_consensus.cc:2781] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:56:35.555431 30265 raft_consensus.cc:482] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:56:35.555475 30265 raft_consensus.cc:3037] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:35.556504 30265 raft_consensus.cc:504] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.556835 30265 leader_election.cc:288] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [CANDIDATE]: Term 1 election: Requested vote from peers fd445070c97e419e97a68d8b5660f316 (127.3.220.2:38141), eea8174356614449aa30ee22d125389f (127.3.220.1:35901)
I20260809 01:56:35.557143 29941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb951d4e0f6648b2b701509a79e88f93" candidate_uuid: "2c02926848e14920af388c429e94af63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd445070c97e419e97a68d8b5660f316"
I20260809 01:56:35.557236 29707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb951d4e0f6648b2b701509a79e88f93" candidate_uuid: "2c02926848e14920af388c429e94af63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea8174356614449aa30ee22d125389f"
I20260809 01:56:35.557256 29941 raft_consensus.cc:3037] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:35.557384 29707 raft_consensus.cc:3037] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:35.558382 29941 raft_consensus.cc:2454] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c02926848e14920af388c429e94af63 in term 1.
I20260809 01:56:35.558640 29707 raft_consensus.cc:2454] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c02926848e14920af388c429e94af63 in term 1.
I20260809 01:56:35.558768 30050 leader_election.cc:302] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c02926848e14920af388c429e94af63, fd445070c97e419e97a68d8b5660f316; no voters:
I20260809 01:56:35.558955 30265 raft_consensus.cc:2781] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:35.559211 30265 raft_consensus.cc:686] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 1 LEADER]: Becoming Leader. State: Replica: 2c02926848e14920af388c429e94af63, State: Running, Role: LEADER
I20260809 01:56:35.559494 30265 consensus_queue.cc:237] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } }
I20260809 01:56:35.562549 29484 catalog_manager.cc:5495] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c02926848e14920af388c429e94af63 (127.3.220.3). New cstate: current_term: 1 leader_uuid: "2c02926848e14920af388c429e94af63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c02926848e14920af388c429e94af63" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 46867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 } health_report { overall_health: UNKNOWN } } }
I20260809 01:56:35.573985 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.570308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.3.220.3:46867)
0809 01:56:35.570449 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:35.570454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:35.570468 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:56:35.570630 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:35.570632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:35.570685 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:35.570797 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:56:35.571059 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.571598 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.572465 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:35.572706 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:35.572831 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:35.573134 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:56:35.573524 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:56:35.573713 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:35.573772 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 01:56:35.573828 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:56:35.579741 29941 raft_consensus.cc:1261] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 1 FOLLOWER]: Refusing update from remote peer 2c02926848e14920af388c429e94af63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:35.579739 29707 raft_consensus.cc:1261] T cb951d4e0f6648b2b701509a79e88f93 P eea8174356614449aa30ee22d125389f [term 1 FOLLOWER]: Refusing update from remote peer 2c02926848e14920af388c429e94af63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:35.580511 30269 consensus_queue.cc:1035] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea8174356614449aa30ee22d125389f" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 35901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:35.580713 30265 consensus_queue.cc:1035] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd445070c97e419e97a68d8b5660f316" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:35.590248 30274 mvcc.cc:205] Tried to move back new op lower bound from 7316441479487733760 to 7316441479413911552. Current Snapshot: MvccSnapshot[applied={T|T < 7316441479487733760}]
I20260809 01:56:35.591044 30275 mvcc.cc:205] Tried to move back new op lower bound from 7316441479487733760 to 7316441479413911552. Current Snapshot: MvccSnapshot[applied={T|T < 7316441479487733760}]
I20260809 01:56:35.589907 30273 mvcc.cc:205] Tried to move back new op lower bound from 7316441479487733760 to 7316441479413911552. Current Snapshot: MvccSnapshot[applied={T|T < 7316441479487733760 or (T in {7316441479487733760})}]
I20260809 01:56:35.597302 29485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:35.597450 29485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:35.599858 29485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e25e84136664a3cb7374fa9f9116db8: Sending DeleteTablet for 3 replicas of tablet cb951d4e0f6648b2b701509a79e88f93
I20260809 01:56:35.600684 30115 tablet_service.cc:1514] Processing DeleteTablet for tablet cb951d4e0f6648b2b701509a79e88f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:35 UTC) from {username='slave'} at 127.0.0.1:50890
I20260809 01:56:35.600677 29910 tablet_service.cc:1514] Processing DeleteTablet for tablet cb951d4e0f6648b2b701509a79e88f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:35 UTC) from {username='slave'} at 127.0.0.1:36698
I20260809 01:56:35.601490 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 29578
I20260809 01:56:35.601495 29679 tablet_service.cc:1514] Processing DeleteTablet for tablet cb951d4e0f6648b2b701509a79e88f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:35 UTC) from {username='slave'} at 127.0.0.1:54256
I20260809 01:56:35.602058 30288 tablet_replica.cc:330] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: stopping tablet replica
I20260809 01:56:35.602308 30289 tablet_replica.cc:330] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: stopping tablet replica
I20260809 01:56:35.602728 30288 raft_consensus.cc:2229] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:56:35.603142 30289 raft_consensus.cc:2229] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:35.603543 30289 raft_consensus.cc:2258] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:35.603997 30288 raft_consensus.cc:2258] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:35.608673 30289 ts_tablet_manager.cc:1905] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:35.609419 30288 ts_tablet_manager.cc:1905] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:35.612180 30288 ts_tablet_manager.cc:1918] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:35.612298 30288 log.cc:1196] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/cb951d4e0f6648b2b701509a79e88f93
I20260809 01:56:35.612510 30289 ts_tablet_manager.cc:1918] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:35.612610 30288 ts_tablet_manager.cc:1939] T cb951d4e0f6648b2b701509a79e88f93 P 2c02926848e14920af388c429e94af63: Deleting consensus metadata
I20260809 01:56:35.612630 30289 log.cc:1196] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/cb951d4e0f6648b2b701509a79e88f93
I20260809 01:56:35.613001 30289 ts_tablet_manager.cc:1939] T cb951d4e0f6648b2b701509a79e88f93 P fd445070c97e419e97a68d8b5660f316: Deleting consensus metadata
I20260809 01:56:35.614156 29463 catalog_manager.cc:4841] TS 2c02926848e14920af388c429e94af63 (127.3.220.3:46867): tablet cb951d4e0f6648b2b701509a79e88f93 (table test-table [id=81ca0d671c894dbfb5d96a2dc79af42a]) successfully deleted
I20260809 01:56:35.614323 29465 catalog_manager.cc:4841] TS fd445070c97e419e97a68d8b5660f316 (127.3.220.2:38141): tablet cb951d4e0f6648b2b701509a79e88f93 (table test-table [id=81ca0d671c894dbfb5d96a2dc79af42a]) successfully deleted
W20260809 01:56:35.616791 29463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:35901 (error 108)
I20260809 01:56:35.617730 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 29808
I20260809 01:56:35.617952 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:35.617622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:35901 (local address 127.0.0.1:54258)
0809 01:56:35.617762 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:35.617846 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:35901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 01:56:35.635856 29463 catalog_manager.cc:4584] TS eea8174356614449aa30ee22d125389f (127.3.220.1:35901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb951d4e0f6648b2b701509a79e88f93: Network error: Client connection negotiation failed: client connection to 127.3.220.1:35901: connect: Connection refused (error 111)
I20260809 01:56:35.641655 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 30021
I20260809 01:56:35.652985 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 29441
2026-08-09T01:56:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1072 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:35 dist-test-slave-gkw7 krb5kdc[30296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:35 dist-test-slave-gkw7 krb5kdc[30296](info): set up 2 sockets
Aug 09 01:56:35 dist-test-slave-gkw7 krb5kdc[30296](info): commencing operation
krb5kdc: starting...
W20260809 01:56:36.897773 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:36 dist-test-slave-gkw7 krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:56:37.028582 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39317
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:39317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:37.143185 30309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:37.143452 30309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:37.143527 30309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:37.148202 30309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:37.148260 30309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:37.148280 30309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:37.148298 30309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:37.148346 30309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:37.153707 30309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:39317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:37.154819 30309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:37.155864 30309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:37.164893 30325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:37.164919 30319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:37.164893 30323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:37.164942 30309 server_base.cc:1034] running on GCE node
I20260809 01:56:37.165975 30309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:37.167162 30309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:37.168388 30309 hybrid_clock.cc:648] HybridClock initialized: now 1786240597168356 us; error 48 us; skew 500 ppm
Aug 09 01:56:37 dist-test-slave-gkw7 krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:37.174264 30309 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:56:37.175212 30309 webserver.cc:460] Webserver started at http://127.3.220.62:37403/ using document root <none> and password file <none>
I20260809 01:56:37.175766 30309 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:37.175827 30309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:37.175995 30309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:37.178174 30309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "518dde0786c04cc6b1826c1676c94d15"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "5d07cf1d1c70b6130b57bbd0fef75b35"
server_key_iv: "7151683b5122ee61bcbe933de5e6b9cb"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.178721 30309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "518dde0786c04cc6b1826c1676c94d15"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "5d07cf1d1c70b6130b57bbd0fef75b35"
server_key_iv: "7151683b5122ee61bcbe933de5e6b9cb"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.182603 30309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 01:56:37.185040 30337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.186347 30309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:56:37.186492 30309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "518dde0786c04cc6b1826c1676c94d15"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "5d07cf1d1c70b6130b57bbd0fef75b35"
server_key_iv: "7151683b5122ee61bcbe933de5e6b9cb"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.186604 30309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:37.200946 30309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:37.203837 30309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:37.204063 30309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:37.212226 30309 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:39317
I20260809 01:56:37.212220 30420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:39317 every 8 connection(s)
I20260809 01:56:37.213363 30309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:37.215610 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 30309
I20260809 01:56:37.215803 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:37.216130 3952 external_mini_cluster.cc:1428] Setting key 772de537365a9c39217d91fad4dd711f
I20260809 01:56:37.216451 30421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:37.223620 30421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15: Bootstrap starting.
Aug 09 01:56:37 dist-test-slave-gkw7 krb5kdc[30296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:37.226532 30421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:37.227483 30421 log.cc:824] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:37.229938 30421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15: No bootstrap required, opened a new log
I20260809 01:56:37.232095 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:37.217727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.3.220.62:39317)
0809 01:56:37.218176 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:37.218193 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:37.218231 (+ 38us) server_negotiation.cc:407] Connection header received
0809 01:56:37.219230 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:37.219251 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:37.219593 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:37.219885 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:37.220879 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.222092 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.223285 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.223834 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.223997 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:37.226908 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:37.226914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:37.226930 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:37.226961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:37.228930 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:37.229370 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:37.229376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:37.229382 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:37.229520 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:37.229840 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:37.229843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:37.229845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:37.230377 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:37.230558 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:37.230816 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 01:56:37.231119 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":150,"threads_started":1}
I20260809 01:56:37.234478 30421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518dde0786c04cc6b1826c1676c94d15" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39317 } }
I20260809 01:56:37.234735 30421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:37.234782 30421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518dde0786c04cc6b1826c1676c94d15, State: Initialized, Role: FOLLOWER
I20260809 01:56:37.235198 30421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518dde0786c04cc6b1826c1676c94d15" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39317 } }
I20260809 01:56:37.235333 30421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:37.235383 30421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:37.235447 30421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:37.236538 30421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518dde0786c04cc6b1826c1676c94d15" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39317 } }
I20260809 01:56:37.236845 30421 leader_election.cc:302] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 518dde0786c04cc6b1826c1676c94d15; no voters:
I20260809 01:56:37.237094 30421 leader_election.cc:288] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:37.237233 30428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:37.237489 30428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [term 1 LEADER]: Becoming Leader. State: Replica: 518dde0786c04cc6b1826c1676c94d15, State: Running, Role: LEADER
I20260809 01:56:37.237818 30428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518dde0786c04cc6b1826c1676c94d15" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39317 } }
I20260809 01:56:37.238374 30421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:37.238950 30428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 518dde0786c04cc6b1826c1676c94d15. Latest consensus state: current_term: 1 leader_uuid: "518dde0786c04cc6b1826c1676c94d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518dde0786c04cc6b1826c1676c94d15" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39317 } } }
I20260809 01:56:37.239068 30428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:37.239744 30430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "518dde0786c04cc6b1826c1676c94d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518dde0786c04cc6b1826c1676c94d15" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39317 } } }
I20260809 01:56:37.239862 30430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:37.240127 30435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:37.243412 30435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:37.248924 30435 catalog_manager.cc:1347] Generated new cluster ID: 1aa886dd09b44a06ae3c6de0217afc14
I20260809 01:56:37.248996 30435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:37.272060 30435 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:37.274240 30435 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:37.284418 30435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15: Generated new TSK 0
I20260809 01:56:37.285187 30435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:56:37.326174 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39317
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:37.447968 30457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:37.448246 30457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:37.448289 30457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:37.452303 30457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:37.452368 30457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:37.452459 30457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:56:37.457798 30457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:37.458892 30457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:37.459969 30457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:37.470189 30472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:37.470369 30457 server_base.cc:1034] running on GCE node
W20260809 01:56:37.470242 30469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:37.470201 30470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:37.470942 30457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:37.471628 30457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:37.472803 30457 hybrid_clock.cc:648] HybridClock initialized: now 1786240597472785 us; error 31 us; skew 500 ppm
Aug 09 01:56:37 dist-test-slave-gkw7 krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:37.475629 30457 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:56:37.476569 30457 webserver.cc:460] Webserver started at http://127.3.220.1:44705/ using document root <none> and password file <none>
I20260809 01:56:37.477219 30457 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:37.477319 30457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:37.477628 30457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:37.479781 30457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "576ae44e47b84a0a9c8e54abe1b6e21d"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "cd31948b8c884a46288c8c2ccf31e67d"
server_key_iv: "81b540c9b60d0f39dd143404e5721226"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.480370 30457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "576ae44e47b84a0a9c8e54abe1b6e21d"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "cd31948b8c884a46288c8c2ccf31e67d"
server_key_iv: "81b540c9b60d0f39dd143404e5721226"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.484311 30457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:56:37.486918 30484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.488168 30457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:56:37.488327 30457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "576ae44e47b84a0a9c8e54abe1b6e21d"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "cd31948b8c884a46288c8c2ccf31e67d"
server_key_iv: "81b540c9b60d0f39dd143404e5721226"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.488444 30457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:37.520155 30457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:37.523252 30457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:37.523473 30457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:37.524183 30457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:37.525053 30457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:37.525130 30457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.525199 30457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:37.525245 30457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.534322 30457 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:34289
I20260809 01:56:37.534353 30669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:34289 every 8 connection(s)
I20260809 01:56:37.535270 30457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:56:37.544459 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 30457
I20260809 01:56:37.544612 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:56:37.545017 3952 external_mini_cluster.cc:1428] Setting key e71bbea1a6a2606c02a6a606e51bcc57
Aug 09 01:56:37 dist-test-slave-gkw7 krb5kdc[30296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:37.550093 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:37.537216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:44325 (local address 127.3.220.62:39317)
0809 01:56:37.537392 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:37.537397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:37.538474 (+ 1077us) server_negotiation.cc:407] Connection header received
0809 01:56:37.539943 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:37.539946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:37.539997 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:37.540078 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:37.541301 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.541909 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.543018 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.543244 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.543321 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:37.546374 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:37.546377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:37.546379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:37.546404 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:37.547893 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:37.548448 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:37.548452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:37.548454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:37.548540 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:37.548896 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:37.548901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:37.548903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:37.549091 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:37.549205 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:37.549785 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 01:56:37.549917 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:56:37.550921 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:37.537764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39317 (local address 127.3.220.1:44325)
0809 01:56:37.538271 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:37.538317 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:37.539723 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:37.540292 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:37.540299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:37.540679 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:37.541162 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:37.541175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.542022 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.542024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:37.542914 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:37.542923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.543385 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.543387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:37.543538 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:37.544106 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:37.544123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:37.546141 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:37.548029 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:37.548035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:37.548050 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:37.548333 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:37.548637 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:37.548642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:37.548644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:37.548796 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:37.549204 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:37.549213 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:37.549471 (+ 258us) client_negotiation.cc:769] Sending connection context
0809 01:56:37.549720 (+ 249us) client_negotiation.cc:240] Negotiation successful
0809 01:56:37.550055 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260809 01:56:37.552255 30673 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39317
I20260809 01:56:37.552527 30673 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:37.553028 30673 heartbeater.cc:511] Master 127.3.220.62:39317 requested a full tablet report, sending...
I20260809 01:56:37.554764 30366 ts_manager.cc:194] Registered new tserver with Master: 576ae44e47b84a0a9c8e54abe1b6e21d (127.3.220.1:34289)
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:56:37.556450 30366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:44325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:56:37.574851 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39317
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:37.690558 30678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:37.690824 30678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:37.690904 30678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:37.695079 30678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:37.695139 30678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:37.695243 30678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:56:37.700563 30678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:37.701714 30678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:37.702817 30678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:37.713042 30686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:37.713115 30683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:37.713044 30690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:37.713351 30678 server_base.cc:1034] running on GCE node
I20260809 01:56:37.713827 30678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:37.714511 30678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:37.715700 30678 hybrid_clock.cc:648] HybridClock initialized: now 1786240597715687 us; error 34 us; skew 500 ppm
Aug 09 01:56:37 dist-test-slave-gkw7 krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:37.718619 30678 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:56:37.719501 30678 webserver.cc:460] Webserver started at http://127.3.220.2:41131/ using document root <none> and password file <none>
I20260809 01:56:37.720072 30678 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:37.720182 30678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:37.720429 30678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:37.722587 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "240f8405c6b44fc8b68cd46f3607f27b"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "a853579e84b2a11301ad98be985e2ea6"
server_key_iv: "6f64397abc9c248388272ce01c2f099f"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.723172 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "240f8405c6b44fc8b68cd46f3607f27b"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "a853579e84b2a11301ad98be985e2ea6"
server_key_iv: "6f64397abc9c248388272ce01c2f099f"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.726915 30678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:56:37.729573 30700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.730779 30678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:56:37.730926 30678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "240f8405c6b44fc8b68cd46f3607f27b"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "a853579e84b2a11301ad98be985e2ea6"
server_key_iv: "6f64397abc9c248388272ce01c2f099f"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.731055 30678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:37.752167 30678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:37.755597 30678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:37.755764 30678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:37.756304 30678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:37.757095 30678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:37.757143 30678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.757184 30678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:37.757198 30678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.766413 30678 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:38153
I20260809 01:56:37.766446 30891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:38153 every 8 connection(s)
I20260809 01:56:37.767309 30678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:56:37.772291 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 30678
I20260809 01:56:37.772437 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:56:37.772728 3952 external_mini_cluster.cc:1428] Setting key 82797db4ae988b392b87b294b274048c
Aug 09 01:56:37 dist-test-slave-gkw7 krb5kdc[30296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:37.782312 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:37.769126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:44595 (local address 127.3.220.62:39317)
0809 01:56:37.769266 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:37.769270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:37.770088 (+ 818us) server_negotiation.cc:407] Connection header received
0809 01:56:37.771408 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:37.771411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:37.771472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:37.771584 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:37.772941 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.773674 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.774993 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.775223 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.775366 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:37.778299 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:37.778302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:37.778304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:37.778327 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:37.779881 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:37.780543 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:37.780545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:37.780546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:37.780621 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:37.781054 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:37.781056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:37.781057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:37.781230 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:37.781304 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:37.782082 (+ 778us) server_negotiation.cc:299] Negotiation successful
0809 01:56:37.782191 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 01:56:37.783581 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:37.769444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39317 (local address 127.3.220.2:44595)
0809 01:56:37.769930 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:37.769975 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:37.771181 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:37.771762 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:37.771770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:37.772166 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:37.772782 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:37.772806 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.773811 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.773820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:37.774854 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:37.774864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.775407 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.775411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:37.775583 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:37.776224 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:37.776240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:37.778112 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:37.780022 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:37.780031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:37.780051 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:37.780441 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:37.780721 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:37.780728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:37.780731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:37.780943 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:37.781348 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:37.781358 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:37.781763 (+ 405us) client_negotiation.cc:769] Sending connection context
0809 01:56:37.782026 (+ 263us) client_negotiation.cc:240] Negotiation successful
0809 01:56:37.782424 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:56:37.785012 30894 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39317
I20260809 01:56:37.785283 30894 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:37.785964 30894 heartbeater.cc:511] Master 127.3.220.62:39317 requested a full tablet report, sending...
I20260809 01:56:37.787039 30366 ts_manager.cc:194] Registered new tserver with Master: 240f8405c6b44fc8b68cd46f3607f27b (127.3.220.2:38153)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:56:37.787684 30366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:44595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:56:37.802719 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39317
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:37.917673 30905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:37.917945 30905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:37.917989 30905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:37.921957 30905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:37.922010 30905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:37.922066 30905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:56:37.927139 30905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:37.928120 30905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:37.929087 30905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:56:37.939411 30905 server_base.cc:1034] running on GCE node
W20260809 01:56:37.939397 30911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:37.939401 30912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:37.939637 30914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:37.940127 30905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:37.940812 30905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:37.942004 30905 hybrid_clock.cc:648] HybridClock initialized: now 1786240597941987 us; error 29 us; skew 500 ppm
Aug 09 01:56:37 dist-test-slave-gkw7 krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:37.944815 30905 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:56:37.945817 30905 webserver.cc:460] Webserver started at http://127.3.220.3:44821/ using document root <none> and password file <none>
I20260809 01:56:37.946409 30905 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:37.946506 30905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:37.946749 30905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:37.948911 30905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "89501ab2f4b6409895b994703bc10817"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "7ff5e6bb5120822396110a4b7097f6c7"
server_key_iv: "07332c34fe33c5ec6ede3c4dd4de3f44"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.949498 30905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "89501ab2f4b6409895b994703bc10817"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "7ff5e6bb5120822396110a4b7097f6c7"
server_key_iv: "07332c34fe33c5ec6ede3c4dd4de3f44"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.953215 30905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:56:37.955739 30925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.956984 30905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:56:37.957130 30905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "89501ab2f4b6409895b994703bc10817"
format_stamp: "Formatted at 2026-08-09 01:56:37 on dist-test-slave-gkw7"
server_key: "7ff5e6bb5120822396110a4b7097f6c7"
server_key_iv: "07332c34fe33c5ec6ede3c4dd4de3f44"
server_key_version: "encryptionkey@0"
I20260809 01:56:37.957247 30905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:37.976598 30905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:37.979628 30905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:37.979840 30905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:37.980460 30905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:37.981313 30905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:37.981386 30905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.981459 30905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:37.981513 30905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:37.991010 30905 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:37963
I20260809 01:56:37.991058 31108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:37963 every 8 connection(s)
I20260809 01:56:37.991914 30905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:56:38.000284 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 30905
I20260809 01:56:38.000452 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:56:38.000856 3952 external_mini_cluster.cc:1428] Setting key 55dfcc917b0aa809bc3b20615abddced
Aug 09 01:56:38 dist-test-slave-gkw7 krb5kdc[30296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:38.006237 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:37.993844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:33433 (local address 127.3.220.62:39317)
0809 01:56:37.993989 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:37.993993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:37.994871 (+ 878us) server_negotiation.cc:407] Connection header received
0809 01:56:37.996202 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:37.996205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:37.996264 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:37.996376 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:37.997612 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.998150 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.999314 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.999540 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.999658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.002850 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:38.002854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:38.002857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:38.002887 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:38.004336 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:38.004816 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:38.004819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:38.004820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:38.004898 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:38.005199 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:38.005201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:38.005202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:38.005381 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:38.005458 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.006008 (+ 550us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.006118 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:56:38.007103 31115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:37.994243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39317 (local address 127.3.220.3:33433)
0809 01:56:37.994720 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:37.994763 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:37.995978 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:37.996564 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:37.996572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:37.996933 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:37.997422 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:37.997437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.998282 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.998286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:37.999178 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:37.999188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:37.999704 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:37.999708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:37.999882 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.000750 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:38.000773 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:38.002659 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:38.004447 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:38.004452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:38.004465 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:38.004721 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:38.004979 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:38.004981 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:38.004983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:38.005117 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:38.005471 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:38.005476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:38.005775 (+ 299us) client_negotiation.cc:769] Sending connection context
0809 01:56:38.005954 (+ 179us) client_negotiation.cc:240] Negotiation successful
0809 01:56:38.006229 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260809 01:56:38.008281 31111 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39317
I20260809 01:56:38.008529 31111 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:38.009037 31111 heartbeater.cc:511] Master 127.3.220.62:39317 requested a full tablet report, sending...
I20260809 01:56:38.010152 30366 ts_manager.cc:194] Registered new tserver with Master: 89501ab2f4b6409895b994703bc10817 (127.3.220.3:37963)
I20260809 01:56:38.010689 30366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:33433
I20260809 01:56:38.014930 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:56:38.024588 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.016946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.3.220.62:39317)
0809 01:56:38.017103 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.017109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.017273 (+ 164us) server_negotiation.cc:407] Connection header received
0809 01:56:38.017430 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.017433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.017478 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.017642 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:38.018419 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.018935 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.019733 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.019947 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.020024 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.021640 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:38.021643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:38.021645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:38.021667 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:38.022926 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:38.023405 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:38.023408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:38.023410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:38.023482 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:38.023749 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:38.023751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:38.023752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:38.023937 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:38.024050 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.024276 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.024394 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 01:56:38.027562 30366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:56:38.030726 30366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:56:38.046166 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.039638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:34289 (local address 127.0.0.1:37158)
0809 01:56:38.040115 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:38.040164 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:38.040302 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:38.041768 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:38.041774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:38.041829 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:38.042078 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.042089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.043091 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.043097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.044394 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.044406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.045586 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.045592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.045728 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.045786 (+ 58us) client_negotiation.cc:769] Sending connection context
0809 01:56:38.045910 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 01:56:38.046035 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":103,"spinlock_wait_cycles":5248,"thread_start_us":81,"threads_started":1}
I20260809 01:56:38.046417 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.039990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.3.220.2:38153)
0809 01:56:38.040625 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.040643 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.041029 (+ 386us) server_negotiation.cc:407] Connection header received
0809 01:56:38.041119 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.041125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.041361 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.041527 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:38.042212 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.043405 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.044923 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.045738 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.045889 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.046027 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.046118 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.046214 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":124,"threads_started":1}
I20260809 01:56:38.046641 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.040397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:37963 (local address 127.0.0.1:45390)
0809 01:56:38.040877 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:38.040894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:38.040992 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:38.041821 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:38.041824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:38.041856 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:38.042065 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.042073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.043584 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.043587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.044744 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.044753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.045645 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.045648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.046282 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.046311 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 01:56:38.046392 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 01:56:38.046458 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":522,"spinlock_wait_cycles":384,"thread_start_us":118,"threads_started":1}
I20260809 01:56:38.047003 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.039821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38153 (local address 127.0.0.1:51450)
0809 01:56:38.040806 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:38.040820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:38.040922 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:38.041758 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:38.041765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:38.041816 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:38.042079 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.042085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.043564 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.043567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.044776 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.044785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.045887 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.045891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.045977 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.045998 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 01:56:38.046074 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 01:56:38.046149 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":99,"spinlock_wait_cycles":16768,"thread_start_us":104,"threads_started":1}
I20260809 01:56:38.047142 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.040810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.3.220.1:34289)
0809 01:56:38.041204 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.041232 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.041262 (+ 30us) server_negotiation.cc:407] Connection header received
0809 01:56:38.041344 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.041349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.041596 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.042047 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:38.042217 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.042955 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.044540 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.045403 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.045522 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.045724 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.046857 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.046947 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":84,"threads_started":1}
I20260809 01:56:38.047610 31136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.040607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.3.220.3:37963)
0809 01:56:38.041417 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.041427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.041446 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:56:38.041510 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.041514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.041708 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.041869 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:38.042212 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.043406 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.044885 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.045452 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.045729 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.045881 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.047337 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.047463 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":143,"threads_started":1}
I20260809 01:56:38.050081 30795 tablet_service.cc:1467] Processing CreateTablet for tablet ce49f26bd1d1482fac929ea73ae15a21 (DEFAULT_TABLE table=test-table [id=5840d943f3d7446d9b4940caa66e9972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:38.050496 31002 tablet_service.cc:1467] Processing CreateTablet for tablet ce49f26bd1d1482fac929ea73ae15a21 (DEFAULT_TABLE table=test-table [id=5840d943f3d7446d9b4940caa66e9972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:38.050496 30560 tablet_service.cc:1467] Processing CreateTablet for tablet ce49f26bd1d1482fac929ea73ae15a21 (DEFAULT_TABLE table=test-table [id=5840d943f3d7446d9b4940caa66e9972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:38.051198 30795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce49f26bd1d1482fac929ea73ae15a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:38.051575 30560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce49f26bd1d1482fac929ea73ae15a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:38.051575 31002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce49f26bd1d1482fac929ea73ae15a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:38.057994 31139 tablet_bootstrap.cc:492] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: Bootstrap starting.
I20260809 01:56:38.058854 31140 tablet_bootstrap.cc:492] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Bootstrap starting.
I20260809 01:56:38.059715 31142 tablet_bootstrap.cc:492] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d: Bootstrap starting.
I20260809 01:56:38.061044 31140 tablet_bootstrap.cc:654] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:38.061235 31139 tablet_bootstrap.cc:654] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:38.061688 31142 tablet_bootstrap.cc:654] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:38.062346 31139 log.cc:824] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:38.062346 31140 log.cc:824] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:38.062399 31142 log.cc:824] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:38.064539 31142 tablet_bootstrap.cc:492] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d: No bootstrap required, opened a new log
I20260809 01:56:38.064644 31140 tablet_bootstrap.cc:492] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: No bootstrap required, opened a new log
I20260809 01:56:38.064644 31139 tablet_bootstrap.cc:492] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: No bootstrap required, opened a new log
I20260809 01:56:38.064765 31142 ts_tablet_manager.cc:1397] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260809 01:56:38.064839 31140 ts_tablet_manager.cc:1397] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 01:56:38.064839 31139 ts_tablet_manager.cc:1397] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 01:56:38.068348 31142 raft_consensus.cc:348] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.068614 31142 raft_consensus.cc:374] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:38.068679 31142 raft_consensus.cc:729] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576ae44e47b84a0a9c8e54abe1b6e21d, State: Initialized, Role: FOLLOWER
I20260809 01:56:38.068768 31139 raft_consensus.cc:348] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.068768 31140 raft_consensus.cc:348] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.069059 31139 raft_consensus.cc:374] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:38.069059 31140 raft_consensus.cc:374] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:38.069118 31139 raft_consensus.cc:729] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240f8405c6b44fc8b68cd46f3607f27b, State: Initialized, Role: FOLLOWER
I20260809 01:56:38.069118 31140 raft_consensus.cc:729] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89501ab2f4b6409895b994703bc10817, State: Initialized, Role: FOLLOWER
I20260809 01:56:38.069205 31142 consensus_queue.cc:260] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.069579 31140 consensus_queue.cc:260] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.069579 31139 consensus_queue.cc:260] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.070583 31139 ts_tablet_manager.cc:1428] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 01:56:38.070590 30673 heartbeater.cc:503] Master 127.3.220.62:39317 was elected leader, sending a full tablet report...
I20260809 01:56:38.070915 31142 ts_tablet_manager.cc:1428] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260809 01:56:38.070955 30894 heartbeater.cc:503] Master 127.3.220.62:39317 was elected leader, sending a full tablet report...
I20260809 01:56:38.071210 31140 ts_tablet_manager.cc:1428] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 01:56:38.071573 31111 heartbeater.cc:503] Master 127.3.220.62:39317 was elected leader, sending a full tablet report...
I20260809 01:56:38.198853 31148 raft_consensus.cc:482] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:38.199096 31148 raft_consensus.cc:504] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.200278 31148 leader_election.cc:288] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 240f8405c6b44fc8b68cd46f3607f27b (127.3.220.2:38153), 89501ab2f4b6409895b994703bc10817 (127.3.220.3:37963)
I20260809 01:56:38.204599 31136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.200814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:41659 (local address 127.3.220.3:37963)
0809 01:56:38.200955 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.200961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.201243 (+ 282us) server_negotiation.cc:407] Connection header received
0809 01:56:38.201427 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.201431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.201483 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.201607 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:38.201954 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.202472 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.203606 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.204019 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.204137 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.204202 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.204389 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.204461 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 01:56:38.204623 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.200813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:58451 (local address 127.3.220.2:38153)
0809 01:56:38.200955 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.200960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.201254 (+ 294us) server_negotiation.cc:407] Connection header received
0809 01:56:38.201446 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.201448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.201490 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.201614 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:38.201969 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.202469 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.203615 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.204029 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.204153 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.204215 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.204422 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.204500 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:56:38.204654 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.200699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:38153 (local address 127.3.220.1:58451)
0809 01:56:38.201145 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:38.201157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:38.201321 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:38.201630 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:38.201633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:38.201672 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:38.201833 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.201850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.202589 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.202593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.203479 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.203488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.204160 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.204164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.204291 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.204308 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 01:56:38.204402 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 01:56:38.204478 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":57,"spinlock_wait_cycles":4224,"thread_start_us":138,"threads_started":1}
I20260809 01:56:38.204658 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.200699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:37963 (local address 127.3.220.1:41659)
0809 01:56:38.201135 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:38.201147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:38.201307 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:38.201625 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:38.201628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:38.201671 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:38.201814 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.201825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.202587 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.202593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.203473 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.203482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.204146 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.204150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.204246 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.204276 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 01:56:38.204371 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 01:56:38.204449 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":69,"spinlock_wait_cycles":30720,"thread_start_us":167,"threads_started":1}
I20260809 01:56:38.205324 30823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce49f26bd1d1482fac929ea73ae15a21" candidate_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240f8405c6b44fc8b68cd46f3607f27b" is_pre_election: true
I20260809 01:56:38.205404 31037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce49f26bd1d1482fac929ea73ae15a21" candidate_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89501ab2f4b6409895b994703bc10817" is_pre_election: true
I20260809 01:56:38.205681 30823 raft_consensus.cc:2454] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 576ae44e47b84a0a9c8e54abe1b6e21d in term 0.
I20260809 01:56:38.205792 31037 raft_consensus.cc:2454] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 576ae44e47b84a0a9c8e54abe1b6e21d in term 0.
I20260809 01:56:38.206321 30496 leader_election.cc:302] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 240f8405c6b44fc8b68cd46f3607f27b, 576ae44e47b84a0a9c8e54abe1b6e21d, 89501ab2f4b6409895b994703bc10817; no voters:
I20260809 01:56:38.206624 31148 raft_consensus.cc:2781] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:56:38.206704 31148 raft_consensus.cc:482] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:56:38.206735 31148 raft_consensus.cc:3037] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:38.207849 31148 raft_consensus.cc:504] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.208235 31148 leader_election.cc:288] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [CANDIDATE]: Term 1 election: Requested vote from peers 240f8405c6b44fc8b68cd46f3607f27b (127.3.220.2:38153), 89501ab2f4b6409895b994703bc10817 (127.3.220.3:37963)
I20260809 01:56:38.208592 30823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce49f26bd1d1482fac929ea73ae15a21" candidate_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240f8405c6b44fc8b68cd46f3607f27b"
I20260809 01:56:38.208597 31037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce49f26bd1d1482fac929ea73ae15a21" candidate_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89501ab2f4b6409895b994703bc10817"
I20260809 01:56:38.208729 30823 raft_consensus.cc:3037] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:38.208738 31037 raft_consensus.cc:3037] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:38.209890 31037 raft_consensus.cc:2454] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 576ae44e47b84a0a9c8e54abe1b6e21d in term 1.
I20260809 01:56:38.209894 30823 raft_consensus.cc:2454] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 576ae44e47b84a0a9c8e54abe1b6e21d in term 1.
I20260809 01:56:38.210246 30489 leader_election.cc:302] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576ae44e47b84a0a9c8e54abe1b6e21d, 89501ab2f4b6409895b994703bc10817; no voters:
I20260809 01:56:38.210431 31148 raft_consensus.cc:2781] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:38.210685 31148 raft_consensus.cc:686] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [term 1 LEADER]: Becoming Leader. State: Replica: 576ae44e47b84a0a9c8e54abe1b6e21d, State: Running, Role: LEADER
I20260809 01:56:38.211050 31148 consensus_queue.cc:237] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } }
I20260809 01:56:38.214644 30360 catalog_manager.cc:5495] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d reported cstate change: term changed from 0 to 1, leader changed from <none> to 576ae44e47b84a0a9c8e54abe1b6e21d (127.3.220.1). New cstate: current_term: 1 leader_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576ae44e47b84a0a9c8e54abe1b6e21d" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 } health_report { overall_health: UNKNOWN } } }
I20260809 01:56:38.237504 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.234006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.3.220.1:34289)
0809 01:56:38.234137 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.234141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.234155 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:56:38.234297 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.234302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.234382 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.234478 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:56:38.234752 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.235258 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.236142 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.236392 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.236474 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.236731 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:56:38.237141 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:56:38.237252 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.237353 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.237389 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 01:56:38.242904 31113 tablet.cc:2369] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:56:38.243875 31037 raft_consensus.cc:1261] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 1 FOLLOWER]: Refusing update from remote peer 576ae44e47b84a0a9c8e54abe1b6e21d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:38.243880 30823 raft_consensus.cc:1261] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 1 FOLLOWER]: Refusing update from remote peer 576ae44e47b84a0a9c8e54abe1b6e21d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:38.244588 31148 consensus_queue.cc:1035] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [LEADER]: Connected to new peer: Peer: permanent_uuid: "89501ab2f4b6409895b994703bc10817" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:38.244845 31154 consensus_queue.cc:1035] T ce49f26bd1d1482fac929ea73ae15a21 P 576ae44e47b84a0a9c8e54abe1b6e21d [LEADER]: Connected to new peer: Peer: permanent_uuid: "240f8405c6b44fc8b68cd46f3607f27b" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:38.253338 31161 mvcc.cc:205] Tried to move back new op lower bound from 7316441490399408128 to 7316441490275299328. Current Snapshot: MvccSnapshot[applied={T|T < 7316441490399408128}]
I20260809 01:56:38.262444 30361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:38.262641 30361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:38.265019 30361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 518dde0786c04cc6b1826c1676c94d15: Sending DeleteTablet for 3 replicas of tablet ce49f26bd1d1482fac929ea73ae15a21
I20260809 01:56:38.265910 31002 tablet_service.cc:1514] Processing DeleteTablet for tablet ce49f26bd1d1482fac929ea73ae15a21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:38 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:45390
I20260809 01:56:38.265939 30795 tablet_service.cc:1514] Processing DeleteTablet for tablet ce49f26bd1d1482fac929ea73ae15a21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:38 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:51450
I20260809 01:56:38.266490 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 30457
I20260809 01:56:38.266755 31171 tablet_replica.cc:330] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: stopping tablet replica
I20260809 01:56:38.267120 31171 raft_consensus.cc:2229] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:38.267460 31171 raft_consensus.cc:2258] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:38.267866 31169 tablet_replica.cc:330] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: stopping tablet replica
I20260809 01:56:38.268159 31169 raft_consensus.cc:2229] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:38.268402 31169 raft_consensus.cc:2258] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:38.270440 31171 ts_tablet_manager.cc:1905] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:38.273636 31171 ts_tablet_manager.cc:1918] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:38.273758 31171 log.cc:1196] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/ce49f26bd1d1482fac929ea73ae15a21
I20260809 01:56:38.274082 31171 ts_tablet_manager.cc:1939] T ce49f26bd1d1482fac929ea73ae15a21 P 89501ab2f4b6409895b994703bc10817: Deleting consensus metadata
I20260809 01:56:38.276096 30340 catalog_manager.cc:4841] TS 89501ab2f4b6409895b994703bc10817 (127.3.220.3:37963): tablet ce49f26bd1d1482fac929ea73ae15a21 (table test-table [id=5840d943f3d7446d9b4940caa66e9972]) successfully deleted
I20260809 01:56:38.277812 31169 ts_tablet_manager.cc:1905] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:38.281329 31169 ts_tablet_manager.cc:1918] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:38.281445 31169 log.cc:1196] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/ce49f26bd1d1482fac929ea73ae15a21
I20260809 01:56:38.281810 31169 ts_tablet_manager.cc:1939] T ce49f26bd1d1482fac929ea73ae15a21 P 240f8405c6b44fc8b68cd46f3607f27b: Deleting consensus metadata
I20260809 01:56:38.283231 30345 catalog_manager.cc:4841] TS 240f8405c6b44fc8b68cd46f3607f27b (127.3.220.2:38153): tablet ce49f26bd1d1482fac929ea73ae15a21 (table test-table [id=5840d943f3d7446d9b4940caa66e9972]) successfully deleted
W20260809 01:56:38.284793 30340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:34289 (error 108)
I20260809 01:56:38.285641 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 30678
I20260809 01:56:38.285915 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.285584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:34289 (local address 127.0.0.1:37176)
0809 01:56:38.285698 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:38.285800 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260809 01:56:38.289093 30340 catalog_manager.cc:4584] TS 576ae44e47b84a0a9c8e54abe1b6e21d (127.3.220.1:34289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce49f26bd1d1482fac929ea73ae15a21: Network error: Client connection negotiation failed: client connection to 127.3.220.1:34289: connect: Connection refused (error 111)
I20260809 01:56:38.299417 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 30905
I20260809 01:56:38.312089 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 30309
2026-08-09T01:56:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2659 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T01:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:38Z Disabled control of system clock
I20260809 01:56:38.347280 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:36831
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:36831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 01:56:38.462775 31177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:38.463057 31177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:38.463095 31177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:38.467167 31177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:38.467231 31177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:38.467247 31177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:38.467263 31177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:38.467278 31177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:38.472234 31177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:36831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:36831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:38.473431 31177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:38.474980 31177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:38.483968 31188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:38.483953 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:38.483953 31187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:38.484326 31177 server_base.cc:1034] running on GCE node
I20260809 01:56:38.485018 31177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:38.486199 31177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:38.487370 31177 hybrid_clock.cc:648] HybridClock initialized: now 1786240598487357 us; error 26 us; skew 500 ppm
I20260809 01:56:38.489298 31177 webserver.cc:460] Webserver started at http://127.3.220.62:40153/ using document root <none> and password file <none>
I20260809 01:56:38.489982 31177 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:38.490049 31177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:38.490317 31177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:38.492466 31177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "f7240b46f992406f87725d75a0a8d67b"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "eabdfaa835926deef450afbdc1547334"
server_key_iv: "5f7a0b4f6e53795d53832047b9e4ab66"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.493053 31177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "f7240b46f992406f87725d75a0a8d67b"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "eabdfaa835926deef450afbdc1547334"
server_key_iv: "5f7a0b4f6e53795d53832047b9e4ab66"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.497040 31177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:56:38.499553 31210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:38.500816 31177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:56:38.500963 31177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "f7240b46f992406f87725d75a0a8d67b"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "eabdfaa835926deef450afbdc1547334"
server_key_iv: "5f7a0b4f6e53795d53832047b9e4ab66"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.501081 31177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:38.515483 31177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:38.516222 31177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:38.516465 31177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:38.523608 31177 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:36831
I20260809 01:56:38.523618 31287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:36831 every 8 connection(s)
I20260809 01:56:38.524572 31177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:38.527666 31288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:38.533226 31288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b: Bootstrap starting.
I20260809 01:56:38.533797 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 31177
I20260809 01:56:38.533929 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:38.534236 3952 external_mini_cluster.cc:1428] Setting key c097d0821fb847c4de7a8597eb7e591e
I20260809 01:56:38.536171 31288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:38.536932 31288 log.cc:824] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:38.538910 31288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b: No bootstrap required, opened a new log
I20260809 01:56:38.542620 31288 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7240b46f992406f87725d75a0a8d67b" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36831 } }
I20260809 01:56:38.542876 31288 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:38.542934 31288 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7240b46f992406f87725d75a0a8d67b, State: Initialized, Role: FOLLOWER
I20260809 01:56:38.543373 31288 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7240b46f992406f87725d75a0a8d67b" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36831 } }
I20260809 01:56:38.543531 31288 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:38.543619 31288 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:38.543747 31288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:38.544056 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.535600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.3.220.62:36831)
0809 01:56:38.536320 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.536346 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.536393 (+ 47us) server_negotiation.cc:407] Connection header received
0809 01:56:38.537052 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.537084 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.537590 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.537936 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:38.538372 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.539420 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.540361 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.540869 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.541047 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.542057 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:38.542062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:38.542076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:38.542473 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:38.542530 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:38.542551 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:38.542669 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.542833 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.543130 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":147,"threads_started":1}
I20260809 01:56:38.544925 31288 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7240b46f992406f87725d75a0a8d67b" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36831 } }
I20260809 01:56:38.545279 31288 leader_election.cc:302] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7240b46f992406f87725d75a0a8d67b; no voters:
I20260809 01:56:38.545667 31288 leader_election.cc:288] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:38.545738 31294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:38.545962 31294 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [term 1 LEADER]: Becoming Leader. State: Replica: f7240b46f992406f87725d75a0a8d67b, State: Running, Role: LEADER
I20260809 01:56:38.546618 31294 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7240b46f992406f87725d75a0a8d67b" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36831 } }
I20260809 01:56:38.547041 31288 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:38.548441 31296 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7240b46f992406f87725d75a0a8d67b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7240b46f992406f87725d75a0a8d67b" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36831 } } }
I20260809 01:56:38.548456 31295 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7240b46f992406f87725d75a0a8d67b. Latest consensus state: current_term: 1 leader_uuid: "f7240b46f992406f87725d75a0a8d67b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7240b46f992406f87725d75a0a8d67b" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 36831 } } }
I20260809 01:56:38.548601 31295 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:38.548601 31296 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:38.549116 31310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:38.552207 31310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:38.557698 31310 catalog_manager.cc:1347] Generated new cluster ID: e658f7e4908143f9810202c51055b014
I20260809 01:56:38.557772 31310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:38.583468 31310 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:38.584512 31310 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:38.591470 31310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b: Generated new TSK 0
I20260809 01:56:38.592197 31310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:38.600253 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36831
--builtin_ntp_servers=127.3.220.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 01:56:38.717718 31323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:38.717978 31323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:38.718014 31323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:38.722087 31323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:38.722139 31323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:38.722193 31323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:56:38.726897 31323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:38.727958 31323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:38.729337 31323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:38.739301 31328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:38.739356 31329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:38.739377 31335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:38.739434 31323 server_base.cc:1034] running on GCE node
I20260809 01:56:38.740015 31323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:38.740654 31323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:38.741854 31323 hybrid_clock.cc:648] HybridClock initialized: now 1786240598741821 us; error 34 us; skew 500 ppm
I20260809 01:56:38.743801 31323 webserver.cc:460] Webserver started at http://127.3.220.1:45713/ using document root <none> and password file <none>
I20260809 01:56:38.744400 31323 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:38.744465 31323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:38.744725 31323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:38.746855 31323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "62ebda2b4a0646898b646ae1063df5fc"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "a593e5c87bcc0a68bfd1fcd2518cf199"
server_key_iv: "9a109c8f407f03485bedf13d73e76676"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.747519 31323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "62ebda2b4a0646898b646ae1063df5fc"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "a593e5c87bcc0a68bfd1fcd2518cf199"
server_key_iv: "9a109c8f407f03485bedf13d73e76676"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.751618 31323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:56:38.754108 31343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:38.755327 31323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:38.755452 31323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "62ebda2b4a0646898b646ae1063df5fc"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "a593e5c87bcc0a68bfd1fcd2518cf199"
server_key_iv: "9a109c8f407f03485bedf13d73e76676"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.755559 31323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:38.772213 31323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:38.772900 31323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:38.773116 31323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:38.773759 31323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:38.774750 31323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:38.774799 31323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:38.774869 31323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:38.774901 31323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:38.783846 31323 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:43349
I20260809 01:56:38.783885 31503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:43349 every 8 connection(s)
I20260809 01:56:38.784768 31323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:56:38.787874 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 31323
I20260809 01:56:38.788026 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:56:38.788321 3952 external_mini_cluster.cc:1428] Setting key 8fb9cfe251e6204295fbd6f87ba6dbb3
I20260809 01:56:38.790349 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36831
--builtin_ntp_servers=127.3.220.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 01:56:38.794634 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.786637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:33645 (local address 127.3.220.62:36831)
0809 01:56:38.786810 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.786813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.787873 (+ 1060us) server_negotiation.cc:407] Connection header received
0809 01:56:38.789581 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.789584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.789639 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.789749 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:38.790756 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.791305 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.792472 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.792703 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.792830 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.794041 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:38.794048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:38.794054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:38.794073 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:38.794101 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:38.794114 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:38.794197 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.794350 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.794423 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 01:56:38.795437 31507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.787047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:36831 (local address 127.3.220.1:33645)
0809 01:56:38.787709 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:38.787750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:38.789343 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:38.789939 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:38.789947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:38.790032 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:38.790610 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.790626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.791446 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.791449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.792339 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.792349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.792873 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.792877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.793046 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.793757 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:38.793759 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:38.793799 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:38.793865 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:38.794204 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:38.794212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:38.794216 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:56:38.794313 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 01:56:38.794543 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260809 01:56:38.796725 31504 heartbeater.cc:347] Connected to a master server at 127.3.220.62:36831
I20260809 01:56:38.796976 31504 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:38.797492 31504 heartbeater.cc:511] Master 127.3.220.62:36831 requested a full tablet report, sending...
I20260809 01:56:38.799005 31240 ts_manager.cc:194] Registered new tserver with Master: 62ebda2b4a0646898b646ae1063df5fc (127.3.220.1:43349)
I20260809 01:56:38.800135 31240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.1:33645
W20260809 01:56:38.906068 31509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:38.906353 31509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:38.906423 31509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:38.910454 31509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:38.910506 31509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:38.910604 31509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:56:38.915347 31509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:38.916494 31509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:38.917953 31509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:38.927800 31520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:38.927909 31515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:38.927800 31516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:38.928011 31509 server_base.cc:1034] running on GCE node
I20260809 01:56:38.928499 31509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:38.929186 31509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:38.930373 31509 hybrid_clock.cc:648] HybridClock initialized: now 1786240598930320 us; error 72 us; skew 500 ppm
I20260809 01:56:38.932276 31509 webserver.cc:460] Webserver started at http://127.3.220.2:34109/ using document root <none> and password file <none>
I20260809 01:56:38.932889 31509 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:38.932982 31509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:38.933228 31509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:38.935391 31509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "7da75b854dae462baa850a2122e05d70"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "f135dc79e7d9f2c2b709566d1a375423"
server_key_iv: "273d11ccf6638853b515a49927f8cbdc"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.935964 31509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "7da75b854dae462baa850a2122e05d70"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "f135dc79e7d9f2c2b709566d1a375423"
server_key_iv: "273d11ccf6638853b515a49927f8cbdc"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.939779 31509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:56:38.942263 31529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:38.943459 31509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:56:38.943596 31509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "7da75b854dae462baa850a2122e05d70"
format_stamp: "Formatted at 2026-08-09 01:56:38 on dist-test-slave-gkw7"
server_key: "f135dc79e7d9f2c2b709566d1a375423"
server_key_iv: "273d11ccf6638853b515a49927f8cbdc"
server_key_version: "encryptionkey@0"
I20260809 01:56:38.943714 31509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:38.963371 31509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:38.964076 31509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:38.964287 31509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:38.964900 31509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:38.965884 31509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:38.965960 31509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:38.966034 31509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:38.966089 31509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:38.975085 31509 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:42305
I20260809 01:56:38.975109 31707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:42305 every 8 connection(s)
I20260809 01:56:38.976040 31509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:56:38.979046 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 31509
I20260809 01:56:38.979179 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:56:38.979537 3952 external_mini_cluster.cc:1428] Setting key db1ff653cdf3d8e89d237c47301d7e09
I20260809 01:56:38.981819 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36831
--builtin_ntp_servers=127.3.220.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 01:56:38.986572 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.977876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:51383 (local address 127.3.220.62:36831)
0809 01:56:38.978055 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:38.978060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:38.979305 (+ 1245us) server_negotiation.cc:407] Connection header received
0809 01:56:38.981083 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:38.981086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:38.981137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:38.981216 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:38.982281 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.982816 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.984423 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.984644 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.984732 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.986110 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:38.986114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:38.986115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:38.986127 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:38.986146 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:38.986154 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:38.986208 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:38.986404 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 01:56:38.986458 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 01:56:38.987859 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:38.978328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:36831 (local address 127.3.220.2:51383)
0809 01:56:38.979124 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:38.979188 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:38.980846 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:38.981396 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:38.981403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:38.981490 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:38.982142 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.982158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.982944 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.982948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.984279 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:38.984293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:38.984805 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:38.984813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:38.985065 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:38.985785 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:38.985790 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:38.985855 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:38.985941 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:38.986236 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:38.986248 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:38.986257 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 01:56:38.986408 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 01:56:38.986737 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260809 01:56:38.989114 31708 heartbeater.cc:347] Connected to a master server at 127.3.220.62:36831
I20260809 01:56:38.989362 31708 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:38.989946 31708 heartbeater.cc:511] Master 127.3.220.62:36831 requested a full tablet report, sending...
I20260809 01:56:38.991044 31240 ts_manager.cc:194] Registered new tserver with Master: 7da75b854dae462baa850a2122e05d70 (127.3.220.2:42305)
I20260809 01:56:38.991613 31240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.2:51383
W20260809 01:56:39.098084 31711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:39.098394 31711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:39.098465 31711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:39.102506 31711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:39.102586 31711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:39.102675 31711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:56:39.107564 31711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:36831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:39.108690 31711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:39.110198 31711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:39.120086 31723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:39.120098 31722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:39.120162 31726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:39.120376 31711 server_base.cc:1034] running on GCE node
I20260809 01:56:39.121057 31711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:39.121791 31711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:39.122999 31711 hybrid_clock.cc:648] HybridClock initialized: now 1786240599122964 us; error 50 us; skew 500 ppm
I20260809 01:56:39.124912 31711 webserver.cc:460] Webserver started at http://127.3.220.3:33577/ using document root <none> and password file <none>
I20260809 01:56:39.125519 31711 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:39.125664 31711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:39.125907 31711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:39.127995 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "33a3f9c38c614193b34eb77c5dd27cfa"
format_stamp: "Formatted at 2026-08-09 01:56:39 on dist-test-slave-gkw7"
server_key: "3a3ee9141e95a53079b22b9ce4a94925"
server_key_iv: "00ed18595527dc74a9cfab4f89cbaf43"
server_key_version: "encryptionkey@0"
I20260809 01:56:39.128575 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "33a3f9c38c614193b34eb77c5dd27cfa"
format_stamp: "Formatted at 2026-08-09 01:56:39 on dist-test-slave-gkw7"
server_key: "3a3ee9141e95a53079b22b9ce4a94925"
server_key_iv: "00ed18595527dc74a9cfab4f89cbaf43"
server_key_version: "encryptionkey@0"
I20260809 01:56:39.132406 31711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260809 01:56:39.135028 31738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:39.136334 31711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:56:39.136462 31711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "33a3f9c38c614193b34eb77c5dd27cfa"
format_stamp: "Formatted at 2026-08-09 01:56:39 on dist-test-slave-gkw7"
server_key: "3a3ee9141e95a53079b22b9ce4a94925"
server_key_iv: "00ed18595527dc74a9cfab4f89cbaf43"
server_key_version: "encryptionkey@0"
I20260809 01:56:39.136571 31711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:39.164680 31711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:39.165442 31711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:39.165697 31711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:39.166368 31711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:39.167379 31711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:39.167428 31711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:39.167500 31711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:39.167538 31711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:39.176239 31711 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:33945
I20260809 01:56:39.176312 31910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:33945 every 8 connection(s)
I20260809 01:56:39.177263 31711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:56:39.180233 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 31711
I20260809 01:56:39.180347 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:56:39.180702 3952 external_mini_cluster.cc:1428] Setting key 1014c33e34bf8f1a539801b6ce83630f
I20260809 01:56:39.186795 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.179228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:36869 (local address 127.3.220.62:36831)
0809 01:56:39.179385 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:39.179389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:39.180611 (+ 1222us) server_negotiation.cc:407] Connection header received
0809 01:56:39.182245 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:39.182248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:39.182305 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:39.182397 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:39.183448 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.183980 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.185091 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.185305 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.185378 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.186386 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:39.186389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:39.186391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:39.186403 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:39.186421 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:39.186428 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:39.186486 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:39.186623 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 01:56:39.186674 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:56:39.187629 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.179603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:36831 (local address 127.3.220.3:36869)
0809 01:56:39.180423 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:39.180482 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:39.182062 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:39.182606 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:39.182615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:39.182727 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:39.183314 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.183330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.184096 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.184099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.184985 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.184994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.185430 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.185433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.185584 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.186147 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:39.186149 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:39.186188 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:39.186249 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:39.186501 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:39.186508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:39.186512 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:56:39.186605 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 01:56:39.186814 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260809 01:56:39.188730 31911 heartbeater.cc:347] Connected to a master server at 127.3.220.62:36831
I20260809 01:56:39.188970 31911 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:39.189491 31911 heartbeater.cc:511] Master 127.3.220.62:36831 requested a full tablet report, sending...
I20260809 01:56:39.190550 31240 ts_manager.cc:194] Registered new tserver with Master: 33a3f9c38c614193b34eb77c5dd27cfa (127.3.220.3:33945)
I20260809 01:56:39.191109 31240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.3:36869
I20260809 01:56:39.194830 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:56:39.200821 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.196742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.3.220.62:36831)
0809 01:56:39.196878 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:39.196883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:39.197074 (+ 191us) server_negotiation.cc:407] Connection header received
0809 01:56:39.197198 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:39.197201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:39.197247 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:39.197345 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:39.197667 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.198169 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.199047 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.199264 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.199380 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.200320 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:39.200323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:39.200325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:39.200337 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:39.200354 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:39.200362 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:39.200501 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:39.200638 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 01:56:39.200676 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 01:56:39.203588 31240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:56:39.206707 31240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:56:39.221446 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.215761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.3.220.2:42305)
0809 01:56:39.216367 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:39.216381 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:39.216407 (+ 26us) server_negotiation.cc:407] Connection header received
0809 01:56:39.216488 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:39.216494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:39.216705 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:39.216876 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:39.217205 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.218516 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.220016 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.220797 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.220907 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.221105 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:39.221226 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 01:56:39.221309 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":108,"threads_started":1}
I20260809 01:56:39.221457 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.215055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:42305 (local address 127.0.0.1:34544)
0809 01:56:39.215512 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:39.215722 (+ 210us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:39.215871 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:39.216873 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:39.216878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:39.216904 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:39.217058 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.217068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.218640 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.218643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.219856 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.219866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.220917 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.220921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.221031 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.221070 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 01:56:39.221186 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 01:56:39.221306 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":52,"thread_start_us":80,"threads_started":1}
I20260809 01:56:39.222234 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.215412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:33945 (local address 127.0.0.1:46176)
0809 01:56:39.216123 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:39.216137 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:39.216247 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:39.217177 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:39.217180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:39.217193 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:39.217282 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.217289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.219814 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.219818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.220590 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.220598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.221673 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.221677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.221887 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.221921 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 01:56:39.221996 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 01:56:39.222082 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":56,"thread_start_us":117,"threads_started":1}
I20260809 01:56:39.222373 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.216150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.3.220.3:33945)
0809 01:56:39.216684 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:39.216699 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:39.216725 (+ 26us) server_negotiation.cc:407] Connection header received
0809 01:56:39.216831 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:39.216836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:39.217023 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:39.217186 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:39.217406 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.218527 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.220720 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.221518 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.221872 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.222068 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:39.222118 (+ 50us) server_negotiation.cc:299] Negotiation successful
0809 01:56:39.222208 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":208,"threads_started":1}
I20260809 01:56:39.222398 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.215761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.3.220.1:43349)
0809 01:56:39.216508 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:39.216519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:39.216539 (+ 20us) server_negotiation.cc:407] Connection header received
0809 01:56:39.216605 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:39.216609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:39.216863 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:39.217335 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:39.217481 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.218272 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.219947 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.220789 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.220907 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.221105 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:39.222159 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0809 01:56:39.222245 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":101,"threads_started":1}
I20260809 01:56:39.222610 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.215130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43349 (local address 127.0.0.1:56402)
0809 01:56:39.215966 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:39.215979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:39.216058 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:39.217019 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:39.217040 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:39.217065 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:39.217180 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.217187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.218398 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.218403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.219820 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.219830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.221673 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.221677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.221858 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.221882 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 01:56:39.221958 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 01:56:39.222029 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":46,"thread_start_us":103,"threads_started":1}
I20260809 01:56:39.225139 31608 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2fdb71f0da4a4aa006094f122754be (DEFAULT_TABLE table=test-table [id=9915af13b863499a943aa5c8124f51c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:39.225353 31813 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2fdb71f0da4a4aa006094f122754be (DEFAULT_TABLE table=test-table [id=9915af13b863499a943aa5c8124f51c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:39.225343 31413 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2fdb71f0da4a4aa006094f122754be (DEFAULT_TABLE table=test-table [id=9915af13b863499a943aa5c8124f51c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:39.226226 31608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2fdb71f0da4a4aa006094f122754be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:39.226419 31413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2fdb71f0da4a4aa006094f122754be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:39.226419 31813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2fdb71f0da4a4aa006094f122754be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:39.233418 31940 tablet_bootstrap.cc:492] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: Bootstrap starting.
I20260809 01:56:39.233994 31941 tablet_bootstrap.cc:492] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc: Bootstrap starting.
I20260809 01:56:39.233992 31942 tablet_bootstrap.cc:492] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: Bootstrap starting.
I20260809 01:56:39.235827 31940 tablet_bootstrap.cc:654] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:39.236204 31941 tablet_bootstrap.cc:654] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:39.236231 31942 tablet_bootstrap.cc:654] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:39.236665 31940 log.cc:824] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:39.237056 31941 log.cc:824] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:39.237056 31942 log.cc:824] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:39.238931 31940 tablet_bootstrap.cc:492] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: No bootstrap required, opened a new log
I20260809 01:56:39.239144 31940 ts_tablet_manager.cc:1397] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260809 01:56:39.239153 31942 tablet_bootstrap.cc:492] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: No bootstrap required, opened a new log
I20260809 01:56:39.239413 31942 ts_tablet_manager.cc:1397] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:56:39.239763 31941 tablet_bootstrap.cc:492] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc: No bootstrap required, opened a new log
I20260809 01:56:39.239959 31941 ts_tablet_manager.cc:1397] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:56:39.242858 31942 raft_consensus.cc:348] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.243100 31942 raft_consensus.cc:374] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:39.243153 31942 raft_consensus.cc:729] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a3f9c38c614193b34eb77c5dd27cfa, State: Initialized, Role: FOLLOWER
I20260809 01:56:39.243610 31942 consensus_queue.cc:260] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.243578 31940 raft_consensus.cc:348] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.243834 31940 raft_consensus.cc:374] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:39.243896 31940 raft_consensus.cc:729] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da75b854dae462baa850a2122e05d70, State: Initialized, Role: FOLLOWER
I20260809 01:56:39.244320 31940 consensus_queue.cc:260] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.244285 31941 raft_consensus.cc:348] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.244521 31941 raft_consensus.cc:374] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:39.244592 31941 raft_consensus.cc:729] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ebda2b4a0646898b646ae1063df5fc, State: Initialized, Role: FOLLOWER
I20260809 01:56:39.244642 31942 ts_tablet_manager.cc:1428] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 01:56:39.245031 31911 heartbeater.cc:503] Master 127.3.220.62:36831 was elected leader, sending a full tablet report...
I20260809 01:56:39.245031 31708 heartbeater.cc:503] Master 127.3.220.62:36831 was elected leader, sending a full tablet report...
I20260809 01:56:39.245049 31941 consensus_queue.cc:260] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.245347 31940 ts_tablet_manager.cc:1428] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260809 01:56:39.245971 31941 ts_tablet_manager.cc:1428] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 01:56:39.246409 31504 heartbeater.cc:503] Master 127.3.220.62:36831 was elected leader, sending a full tablet report...
W20260809 01:56:39.285974 31505 tablet.cc:2369] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:56:39.345920 31948 raft_consensus.cc:482] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:39.346143 31948 raft_consensus.cc:504] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.347543 31948 leader_election.cc:288] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62ebda2b4a0646898b646ae1063df5fc (127.3.220.1:43349), 33a3f9c38c614193b34eb77c5dd27cfa (127.3.220.3:33945)
I20260809 01:56:39.351643 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.347791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43349 (local address 127.3.220.2:55039)
0809 01:56:39.347950 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:39.347969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:39.348084 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:39.348414 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:39.348416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:39.348438 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:39.348599 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.348606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.349551 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.349555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.350462 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.350471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.351225 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.351229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.351328 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.351355 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:56:39.351429 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 01:56:39.351496 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"spinlock_wait_cycles":12672}
I20260809 01:56:39.351670 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.347921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:55039 (local address 127.3.220.1:43349)
0809 01:56:39.348064 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:39.348068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:39.348084 (+ 16us) server_negotiation.cc:407] Connection header received
0809 01:56:39.348223 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:39.348229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:39.348300 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:39.348388 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:39.348702 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.349424 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.350585 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.351086 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.351166 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.351214 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:39.351466 (+ 252us) server_negotiation.cc:299] Negotiation successful
0809 01:56:39.351534 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 01:56:39.351948 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.347803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:33945 (local address 127.3.220.2:36061)
0809 01:56:39.348386 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:39.348400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:39.348504 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:39.348782 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:39.348785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:39.348803 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:39.348933 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.348939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.349885 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.349889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.350748 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:39.350759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.351491 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.351495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:39.351584 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.351606 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 01:56:39.351687 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 01:56:39.351759 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":176,"thread_start_us":153,"threads_started":1}
I20260809 01:56:39.351961 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.347987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:36061 (local address 127.3.220.3:33945)
0809 01:56:39.348165 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:39.348172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:39.348483 (+ 311us) server_negotiation.cc:407] Connection header received
0809 01:56:39.348605 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:39.348609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:39.348669 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:39.348760 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:39.349097 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.349781 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.350872 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.351342 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.351451 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.351504 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:39.351738 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 01:56:39.351837 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 01:56:39.352370 31444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2fdb71f0da4a4aa006094f122754be" candidate_uuid: "7da75b854dae462baa850a2122e05d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ebda2b4a0646898b646ae1063df5fc" is_pre_election: true
I20260809 01:56:39.352653 31842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2fdb71f0da4a4aa006094f122754be" candidate_uuid: "7da75b854dae462baa850a2122e05d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" is_pre_election: true
I20260809 01:56:39.352680 31444 raft_consensus.cc:2454] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da75b854dae462baa850a2122e05d70 in term 0.
I20260809 01:56:39.352914 31842 raft_consensus.cc:2454] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da75b854dae462baa850a2122e05d70 in term 0.
I20260809 01:56:39.353260 31535 leader_election.cc:302] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62ebda2b4a0646898b646ae1063df5fc, 7da75b854dae462baa850a2122e05d70; no voters:
I20260809 01:56:39.353607 31948 raft_consensus.cc:2781] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:56:39.353718 31948 raft_consensus.cc:482] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:56:39.353783 31948 raft_consensus.cc:3037] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:39.354826 31948 raft_consensus.cc:504] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.355176 31948 leader_election.cc:288] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [CANDIDATE]: Term 1 election: Requested vote from peers 62ebda2b4a0646898b646ae1063df5fc (127.3.220.1:43349), 33a3f9c38c614193b34eb77c5dd27cfa (127.3.220.3:33945)
I20260809 01:56:39.355522 31842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2fdb71f0da4a4aa006094f122754be" candidate_uuid: "7da75b854dae462baa850a2122e05d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a3f9c38c614193b34eb77c5dd27cfa"
I20260809 01:56:39.355561 31444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2fdb71f0da4a4aa006094f122754be" candidate_uuid: "7da75b854dae462baa850a2122e05d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ebda2b4a0646898b646ae1063df5fc"
I20260809 01:56:39.355651 31842 raft_consensus.cc:3037] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:39.355665 31444 raft_consensus.cc:3037] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:39.356927 31842 raft_consensus.cc:2454] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da75b854dae462baa850a2122e05d70 in term 1.
I20260809 01:56:39.356927 31444 raft_consensus.cc:2454] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da75b854dae462baa850a2122e05d70 in term 1.
I20260809 01:56:39.357273 31531 leader_election.cc:302] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a3f9c38c614193b34eb77c5dd27cfa, 7da75b854dae462baa850a2122e05d70; no voters:
I20260809 01:56:39.357476 31948 raft_consensus.cc:2781] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:39.357752 31948 raft_consensus.cc:686] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 1 LEADER]: Becoming Leader. State: Replica: 7da75b854dae462baa850a2122e05d70, State: Running, Role: LEADER
I20260809 01:56:39.358119 31948 consensus_queue.cc:237] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } }
I20260809 01:56:39.361443 31239 catalog_manager.cc:5495] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da75b854dae462baa850a2122e05d70 (127.3.220.2). New cstate: current_term: 1 leader_uuid: "7da75b854dae462baa850a2122e05d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da75b854dae462baa850a2122e05d70" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 42305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 } health_report { overall_health: UNKNOWN } } }
I20260809 01:56:39.375654 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.372322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.3.220.2:42305)
0809 01:56:39.372433 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:39.372437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:39.372450 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:56:39.372602 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:39.372604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:39.372658 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:39.372745 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:56:39.372967 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.373464 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.374338 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:39.374544 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:39.374621 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:39.374869 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:56:39.375254 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:56:39.375369 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:39.375441 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 01:56:39.375472 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 01:56:39.381574 31842 raft_consensus.cc:1261] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 1 FOLLOWER]: Refusing update from remote peer 7da75b854dae462baa850a2122e05d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:39.381572 31444 raft_consensus.cc:1261] T 9a2fdb71f0da4a4aa006094f122754be P 62ebda2b4a0646898b646ae1063df5fc [term 1 FOLLOWER]: Refusing update from remote peer 7da75b854dae462baa850a2122e05d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:39.382336 31951 consensus_queue.cc:1035] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a3f9c38c614193b34eb77c5dd27cfa" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 33945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:39.382486 31948 consensus_queue.cc:1035] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ebda2b4a0646898b646ae1063df5fc" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 43349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:39.393173 31961 mvcc.cc:205] Tried to move back new op lower bound from 7316441495059132416 to 7316441494973493248. Current Snapshot: MvccSnapshot[applied={T|T < 7316441495059132416}]
I20260809 01:56:39.397984 31237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:39.398160 31237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:39.400539 31237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7240b46f992406f87725d75a0a8d67b: Sending DeleteTablet for 3 replicas of tablet 9a2fdb71f0da4a4aa006094f122754be
I20260809 01:56:39.401261 31413 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2fdb71f0da4a4aa006094f122754be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:39 UTC) from {username='slave'} at 127.0.0.1:56402
I20260809 01:56:39.401253 31608 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2fdb71f0da4a4aa006094f122754be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:39 UTC) from {username='slave'} at 127.0.0.1:34544
I20260809 01:56:39.402066 31813 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2fdb71f0da4a4aa006094f122754be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:39 UTC) from {username='slave'} at 127.0.0.1:46176
I20260809 01:56:39.402501 31976 tablet_replica.cc:330] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: stopping tablet replica
I20260809 01:56:39.402504 31972 tablet_replica.cc:330] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: stopping tablet replica
I20260809 01:56:39.402734 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 31323
I20260809 01:56:39.402948 31976 raft_consensus.cc:2229] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:39.403290 31976 raft_consensus.cc:2258] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:39.405138 31976 ts_tablet_manager.cc:1905] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:39.409173 31976 ts_tablet_manager.cc:1918] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:39.409405 31976 log.cc:1196] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/9a2fdb71f0da4a4aa006094f122754be
I20260809 01:56:39.409878 31976 ts_tablet_manager.cc:1939] T 9a2fdb71f0da4a4aa006094f122754be P 33a3f9c38c614193b34eb77c5dd27cfa: Deleting consensus metadata
I20260809 01:56:39.402797 31972 raft_consensus.cc:2229] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:56:39.410290 31972 raft_consensus.cc:2258] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:39.411970 31212 catalog_manager.cc:4841] TS 33a3f9c38c614193b34eb77c5dd27cfa (127.3.220.3:33945): tablet 9a2fdb71f0da4a4aa006094f122754be (table test-table [id=9915af13b863499a943aa5c8124f51c7]) successfully deleted
I20260809 01:56:39.412786 31972 ts_tablet_manager.cc:1905] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:39.415669 31972 ts_tablet_manager.cc:1918] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:39.415781 31972 log.cc:1196] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/9a2fdb71f0da4a4aa006094f122754be
I20260809 01:56:39.416123 31972 ts_tablet_manager.cc:1939] T 9a2fdb71f0da4a4aa006094f122754be P 7da75b854dae462baa850a2122e05d70: Deleting consensus metadata
I20260809 01:56:39.417402 31211 catalog_manager.cc:4841] TS 7da75b854dae462baa850a2122e05d70 (127.3.220.2:42305): tablet 9a2fdb71f0da4a4aa006094f122754be (table test-table [id=9915af13b863499a943aa5c8124f51c7]) successfully deleted
W20260809 01:56:39.419549 31216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:43349 (error 108)
I20260809 01:56:39.419659 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 31509
I20260809 01:56:39.422843 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:39.422467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:43349 (local address 127.0.0.1:56408)
0809 01:56:39.422647 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:39.422734 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:43349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 01:56:39.423025 31216 catalog_manager.cc:4584] TS 62ebda2b4a0646898b646ae1063df5fc (127.3.220.1:43349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a2fdb71f0da4a4aa006094f122754be: Network error: Client connection negotiation failed: client connection to 127.3.220.1:43349: connect: Connection refused (error 111)
I20260809 01:56:39.429900 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 31711
I20260809 01:56:39.440313 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 31177
2026-08-09T01:56:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1132 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:39 dist-test-slave-gkw7 krb5kdc[31980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:39 dist-test-slave-gkw7 krb5kdc[31980](info): set up 2 sockets
Aug 09 01:56:39 dist-test-slave-gkw7 krb5kdc[31980](info): commencing operation
krb5kdc: starting...
W20260809 01:56:40.681929 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:40 dist-test-slave-gkw7 krb5kdc[31980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:56:40.816344 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39977
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:39977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:40.930727 32005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:40.931003 32005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:40.931043 32005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:40.935292 32005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:40.935348 32005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:40.935369 32005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:40.935385 32005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:40.935422 32005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:40.940757 32005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:39977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:40.941922 32005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:40.943027 32005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:40.951848 32015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:40.951875 32018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:40.951879 32014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:40.952152 32005 server_base.cc:1034] running on GCE node
I20260809 01:56:40.952703 32005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:40.954008 32005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:40.955174 32005 hybrid_clock.cc:648] HybridClock initialized: now 1786240600955159 us; error 27 us; skew 500 ppm
Aug 09 01:56:40 dist-test-slave-gkw7 krb5kdc[31980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240600, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:40.957979 32005 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:56:40.958871 32005 webserver.cc:460] Webserver started at http://127.3.220.62:33093/ using document root <none> and password file <none>
I20260809 01:56:40.959458 32005 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:40.959553 32005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:40.959792 32005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:40.961928 32005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "bdf60606945641ab9560adbbbf9a787f"
format_stamp: "Formatted at 2026-08-09 01:56:40 on dist-test-slave-gkw7"
server_key: "40aa05805533e7f033f944c09d06e542"
server_key_iv: "a58a00ed75d27617bfbc1ee5878667e5"
server_key_version: "encryptionkey@0"
I20260809 01:56:40.962502 32005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "bdf60606945641ab9560adbbbf9a787f"
format_stamp: "Formatted at 2026-08-09 01:56:40 on dist-test-slave-gkw7"
server_key: "40aa05805533e7f033f944c09d06e542"
server_key_iv: "a58a00ed75d27617bfbc1ee5878667e5"
server_key_version: "encryptionkey@0"
I20260809 01:56:40.966251 32005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:56:40.968734 32026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:40.970165 32005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:56:40.970340 32005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "bdf60606945641ab9560adbbbf9a787f"
format_stamp: "Formatted at 2026-08-09 01:56:40 on dist-test-slave-gkw7"
server_key: "40aa05805533e7f033f944c09d06e542"
server_key_iv: "a58a00ed75d27617bfbc1ee5878667e5"
server_key_version: "encryptionkey@0"
I20260809 01:56:40.970471 32005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:41.002599 32005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:41.005749 32005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:41.005992 32005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:41.014135 32005 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:39977
I20260809 01:56:41.014161 32099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:39977 every 8 connection(s)
I20260809 01:56:41.015214 32005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:41.018173 32100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:41.023839 32100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f: Bootstrap starting.
I20260809 01:56:41.024398 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 32005
I20260809 01:56:41.024571 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:41.024890 3952 external_mini_cluster.cc:1428] Setting key 6a802faa7f19cdda19d36eeab72ccf68
I20260809 01:56:41.026857 32100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:41.027789 32100 log.cc:824] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:41.029834 32100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f: No bootstrap required, opened a new log
I20260809 01:56:41.033273 32100 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf60606945641ab9560adbbbf9a787f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39977 } }
I20260809 01:56:41.033535 32100 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:41.033664 32100 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf60606945641ab9560adbbbf9a787f, State: Initialized, Role: FOLLOWER
I20260809 01:56:41.034220 32100 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf60606945641ab9560adbbbf9a787f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39977 } }
Aug 09 01:56:41 dist-test-slave-gkw7 krb5kdc[31980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:41.034384 32100 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:41.034448 32100 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:41.034552 32100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:41.035558 32100 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf60606945641ab9560adbbbf9a787f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39977 } }
I20260809 01:56:41.035936 32100 leader_election.cc:302] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf60606945641ab9560adbbbf9a787f; no voters:
I20260809 01:56:41.036339 32100 leader_election.cc:288] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:41.036458 32106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:41.036677 32106 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [term 1 LEADER]: Becoming Leader. State: Replica: bdf60606945641ab9560adbbbf9a787f, State: Running, Role: LEADER
I20260809 01:56:41.037011 32106 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf60606945641ab9560adbbbf9a787f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39977 } }
I20260809 01:56:41.037796 32100 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:41.038425 32106 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf60606945641ab9560adbbbf9a787f. Latest consensus state: current_term: 1 leader_uuid: "bdf60606945641ab9560adbbbf9a787f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf60606945641ab9560adbbbf9a787f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39977 } } }
I20260809 01:56:41.038558 32106 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:41.039005 32118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:41.039044 32107 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf60606945641ab9560adbbbf9a787f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf60606945641ab9560adbbbf9a787f" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39977 } } }
I20260809 01:56:41.039279 32107 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:41.042465 32118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:41.042482 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.026382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.3.220.62:39977)
0809 01:56:41.026878 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:41.026900 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:41.026947 (+ 47us) server_negotiation.cc:407] Connection header received
0809 01:56:41.027840 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:41.027867 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:41.028307 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:41.028665 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:41.029517 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.030625 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.031519 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.032019 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.032214 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.035063 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:41.035069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:41.035082 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:41.035110 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:41.038464 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.039049 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.039055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.039061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.039200 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.039540 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.039543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.039545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.040059 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:41.041001 (+ 942us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:41.041167 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 01:56:41.041485 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":118,"threads_started":1}
I20260809 01:56:41.048478 32118 catalog_manager.cc:1347] Generated new cluster ID: aa03920cb5b84369af4b085f044acaf7
I20260809 01:56:41.048556 32118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:41.063182 32118 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:41.064177 32118 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:41.073823 32118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f: Generated new TSK 0
I20260809 01:56:41.074724 32118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.220.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
Entry for principal kudu/127.3.220.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab.
I20260809 01:56:41.109079 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39977
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:41.226655 32140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:41.226939 32140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:41.226984 32140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:41.231014 32140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:41.231081 32140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:41.231151 32140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:56:41.236394 32140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.1.keytab
--principal=kudu/127.3.220.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:41.237557 32140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:41.238772 32140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:56:41.249059 32140 server_base.cc:1034] running on GCE node
W20260809 01:56:41.249040 32147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:41.249116 32152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:41.249042 32149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:41.249840 32140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:41.250451 32140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:41.251646 32140 hybrid_clock.cc:648] HybridClock initialized: now 1786240601251631 us; error 27 us; skew 500 ppm
Aug 09 01:56:41 dist-test-slave-gkw7 krb5kdc[31980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:41.254626 32140 init.cc:377] Logged in from keytab as kudu/127.3.220.1@KRBTEST.COM (short username kudu)
I20260809 01:56:41.255539 32140 webserver.cc:460] Webserver started at http://127.3.220.1:38577/ using document root <none> and password file <none>
I20260809 01:56:41.256126 32140 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:41.256189 32140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:41.256435 32140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:41.258824 32140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "206b3892efef4111b920815492b264a1"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "f30c8a9401d18e47ee77f6fae146cff9"
server_key_iv: "f4cb8b606cc0de7f3d340785aa26cf13"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.259420 32140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "206b3892efef4111b920815492b264a1"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "f30c8a9401d18e47ee77f6fae146cff9"
server_key_iv: "f4cb8b606cc0de7f3d340785aa26cf13"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.263345 32140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:56:41.265908 32161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.267105 32140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:56:41.267232 32140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "206b3892efef4111b920815492b264a1"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "f30c8a9401d18e47ee77f6fae146cff9"
server_key_iv: "f4cb8b606cc0de7f3d340785aa26cf13"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.267347 32140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:41.286798 32140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:41.289817 32140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:41.290021 32140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:41.290644 32140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:41.291503 32140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:41.291553 32140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.291622 32140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:41.291654 32140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.300930 32140 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:44917
I20260809 01:56:41.300954 32319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:44917 every 8 connection(s)
I20260809 01:56:41.301980 32140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:56:41.306680 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 32140
I20260809 01:56:41.306839 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:56:41.307194 3952 external_mini_cluster.cc:1428] Setting key d926a0be2bfba46dc45ddcd0cb6ce5d3
Aug 09 01:56:41 dist-test-slave-gkw7 krb5kdc[31980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.1@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:41.316932 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.303825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:34477 (local address 127.3.220.62:39977)
0809 01:56:41.304006 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:41.304011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:41.304882 (+ 871us) server_negotiation.cc:407] Connection header received
0809 01:56:41.305825 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:41.305828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:41.305877 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:41.305955 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:41.307544 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.308206 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.309865 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.310097 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.310181 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.313208 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:41.313212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:41.313214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:41.313239 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:41.314745 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.315284 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.315287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.315289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.315385 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.315732 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.315737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.315739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.315947 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:41.316055 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:41.316604 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 01:56:41.316739 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 01:56:41.317840 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.304241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39977 (local address 127.3.220.1:34477)
0809 01:56:41.304710 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:41.304755 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:41.305617 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:41.306123 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:41.306130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:41.306612 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:41.307367 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.307389 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.308339 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.308343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.309704 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.309719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.310256 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.310263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.310501 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.311255 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:41.311271 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:41.313029 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:41.314887 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:41.314893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:41.314915 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:41.315173 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:41.315481 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:41.315485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:41.315487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:41.315632 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:41.316071 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:41.316079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:41.316359 (+ 280us) client_negotiation.cc:769] Sending connection context
0809 01:56:41.316563 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 01:56:41.316902 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.2@KRBTEST.COM; defaulting to no policy
I20260809 01:56:41.319222 32320 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39977
I20260809 01:56:41.319459 32320 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:41.319947 32320 heartbeater.cc:511] Master 127.3.220.62:39977 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.2@KRBTEST.COM" created.
I20260809 01:56:41.321655 32048 ts_manager.cc:194] Registered new tserver with Master: 206b3892efef4111b920815492b264a1 (127.3.220.1:44917)
I20260809 01:56:41.322815 32048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.1@KRBTEST.COM'} at 127.3.220.1:34477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
Entry for principal kudu/127.3.220.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab.
I20260809 01:56:41.337047 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39977
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:41.453066 32330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:41.453332 32330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:41.453374 32330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:41.457576 32330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:41.457639 32330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:41.457756 32330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:56:41.463230 32330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.2.keytab
--principal=kudu/127.3.220.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:41.464313 32330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:41.465632 32330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:41.475924 32340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:41.475920 32344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:41.475971 32338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:41.476233 32330 server_base.cc:1034] running on GCE node
I20260809 01:56:41.476686 32330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:41.477335 32330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:41.478528 32330 hybrid_clock.cc:648] HybridClock initialized: now 1786240601478502 us; error 44 us; skew 500 ppm
Aug 09 01:56:41 dist-test-slave-gkw7 krb5kdc[31980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:41.481369 32330 init.cc:377] Logged in from keytab as kudu/127.3.220.2@KRBTEST.COM (short username kudu)
I20260809 01:56:41.482349 32330 webserver.cc:460] Webserver started at http://127.3.220.2:34131/ using document root <none> and password file <none>
I20260809 01:56:41.482882 32330 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:41.482944 32330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:41.483116 32330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:41.485164 32330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "e19cd380c8d97e6fd15943ddd4a0de07"
server_key_iv: "cf6f6436d1f3eac3b8f35e3f4793c79a"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.485769 32330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "e19cd380c8d97e6fd15943ddd4a0de07"
server_key_iv: "cf6f6436d1f3eac3b8f35e3f4793c79a"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.489501 32330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 01:56:41.492486 32352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.493800 32330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:56:41.493907 32330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "e19cd380c8d97e6fd15943ddd4a0de07"
server_key_iv: "cf6f6436d1f3eac3b8f35e3f4793c79a"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.493995 32330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:41.504407 32330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:41.507715 32330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:41.507929 32330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:41.508623 32330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:41.509461 32330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:41.509509 32330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.509617 32330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:41.509651 32330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.518538 32330 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:43365
I20260809 01:56:41.518561 32524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:43365 every 8 connection(s)
I20260809 01:56:41.519420 32330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:56:41.524698 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 32330
I20260809 01:56:41.524863 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:56:41.525205 3952 external_mini_cluster.cc:1428] Setting key cbb6f9aae2f35445fb7369f7fe8af42d
Aug 09 01:56:41 dist-test-slave-gkw7 krb5kdc[31980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.2@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:41.533864 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.521257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:41745 (local address 127.3.220.62:39977)
0809 01:56:41.521416 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:41.521420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:41.522343 (+ 923us) server_negotiation.cc:407] Connection header received
0809 01:56:41.523173 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:41.523176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:41.523225 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:41.523305 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:41.524441 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.525067 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.526545 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.526828 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.526914 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.530166 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:41.530170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:41.530172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:41.530196 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:41.531728 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.532279 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.532283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.532284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.532361 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.532717 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.532721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.532723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.532912 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:41.533024 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:41.533563 (+ 539us) server_negotiation.cc:299] Negotiation successful
0809 01:56:41.533688 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 01:56:41.534654 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.521716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39977 (local address 127.3.220.2:41745)
0809 01:56:41.522181 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:41.522223 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:41.522978 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:41.523485 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:41.523491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:41.523832 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:41.524300 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.524313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.525192 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.525195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.526383 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.526418 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.526984 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.526989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.527204 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.527835 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:41.527861 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:41.529990 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:41.531861 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:41.531867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:41.531883 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:41.532150 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:41.532458 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:41.532462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:41.532464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:41.532610 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:41.533024 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:41.533033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:41.533283 (+ 250us) client_negotiation.cc:769] Sending connection context
0809 01:56:41.533471 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 01:56:41.533833 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.3.220.3@KRBTEST.COM; defaulting to no policy
I20260809 01:56:41.535889 32527 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39977
I20260809 01:56:41.536163 32527 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:41.536674 32527 heartbeater.cc:511] Master 127.3.220.62:39977 requested a full tablet report, sending...
I20260809 01:56:41.537886 32048 ts_manager.cc:194] Registered new tserver with Master: a10f2d9fead64d6ab4e2a7cfcf289d93 (127.3.220.2:43365)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.3@KRBTEST.COM" created.
I20260809 01:56:41.538625 32048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.2@KRBTEST.COM'} at 127.3.220.2:41745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
Entry for principal kudu/127.3.220.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab.
I20260809 01:56:41.554742 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39977
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:41.668746 32535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:41.669008 32535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:41.669051 32535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:41.674310 32535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:41.674382 32535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:41.674443 32535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:56:41.679801 32535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.3.keytab
--principal=kudu/127.3.220.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:39977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:41.680828 32535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:41.681948 32535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:41.691877 32545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:41.691896 32546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:41.691896 32548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:41.692130 32535 server_base.cc:1034] running on GCE node
I20260809 01:56:41.692632 32535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:41.693337 32535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:41.694504 32535 hybrid_clock.cc:648] HybridClock initialized: now 1786240601694502 us; error 49 us; skew 500 ppm
Aug 09 01:56:41 dist-test-slave-gkw7 krb5kdc[31980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:41.697242 32535 init.cc:377] Logged in from keytab as kudu/127.3.220.3@KRBTEST.COM (short username kudu)
I20260809 01:56:41.698320 32535 webserver.cc:460] Webserver started at http://127.3.220.3:43481/ using document root <none> and password file <none>
I20260809 01:56:41.698920 32535 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:41.699016 32535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:41.699257 32535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:41.701377 32535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "418b468ec74446a08bf6a14f64bcaba3"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "017bec8684d3c7e4686b5ed1c359b8b2"
server_key_iv: "08ca8606e19935f04598be607804fbba"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.702051 32535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "418b468ec74446a08bf6a14f64bcaba3"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "017bec8684d3c7e4686b5ed1c359b8b2"
server_key_iv: "08ca8606e19935f04598be607804fbba"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.705931 32535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:56:41.708389 32557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.709656 32535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:41.709822 32535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "418b468ec74446a08bf6a14f64bcaba3"
format_stamp: "Formatted at 2026-08-09 01:56:41 on dist-test-slave-gkw7"
server_key: "017bec8684d3c7e4686b5ed1c359b8b2"
server_key_iv: "08ca8606e19935f04598be607804fbba"
server_key_version: "encryptionkey@0"
I20260809 01:56:41.709944 32535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:41.726543 32535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:41.729419 32535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:41.729667 32535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:41.730283 32535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:41.731145 32535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:41.731220 32535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.731292 32535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:41.731328 32535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:41.740281 32535 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:45581
I20260809 01:56:41.740304 32735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:45581 every 8 connection(s)
I20260809 01:56:41.741170 32535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:56:41.741931 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 32535
I20260809 01:56:41.742085 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:56:41.742398 3952 external_mini_cluster.cc:1428] Setting key 2b51c6acaef9edce424174fbe9739298
Aug 09 01:56:41 dist-test-slave-gkw7 krb5kdc[31980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.3@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:56:41.754344 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.743272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:47901 (local address 127.3.220.62:39977)
0809 01:56:41.743541 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:41.743546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:41.744484 (+ 938us) server_negotiation.cc:407] Connection header received
0809 01:56:41.745463 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:41.745465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:41.745512 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:41.745640 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:41.746768 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.747287 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.748361 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.748568 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.748641 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.751091 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:41.751094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:41.751096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:41.751120 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:41.752550 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.753000 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.753002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.753004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.753076 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.753361 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.753363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.753364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.753597 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:41.753677 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:41.754130 (+ 453us) server_negotiation.cc:299] Negotiation successful
0809 01:56:41.754226 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260809 01:56:41.755383 32740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.743757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:39977 (local address 127.3.220.3:47901)
0809 01:56:41.744298 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:41.744366 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:41.745261 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:41.745806 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:41.745813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:41.746156 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:41.746645 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.746658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.747397 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.747399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.748259 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.748267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.748692 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.748695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.748828 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.749401 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:56:41.749417 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:41.750938 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:41.752647 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:41.752651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:41.752662 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:41.752915 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:41.753153 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:56:41.753156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:56:41.753157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:56:41.753284 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:56:41.753688 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:41.753693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:41.753925 (+ 232us) client_negotiation.cc:769] Sending connection context
0809 01:56:41.754081 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 01:56:41.754320 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":120,"threads_started":1}
I20260809 01:56:41.756496 32737 heartbeater.cc:347] Connected to a master server at 127.3.220.62:39977
I20260809 01:56:41.756740 32737 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:41.757236 32737 heartbeater.cc:511] Master 127.3.220.62:39977 requested a full tablet report, sending...
I20260809 01:56:41.758335 32048 ts_manager.cc:194] Registered new tserver with Master: 418b468ec74446a08bf6a14f64bcaba3 (127.3.220.3:45581)
I20260809 01:56:41.758908 32048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.220.3@KRBTEST.COM'} at 127.3.220.3:47901
I20260809 01:56:41.768376 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:56:41.778251 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.770375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.3.220.62:39977)
0809 01:56:41.770536 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:41.770541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:41.770685 (+ 144us) server_negotiation.cc:407] Connection header received
0809 01:56:41.770830 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:41.770833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:41.770882 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:41.770967 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:41.771735 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.772345 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.773196 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.773422 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.773531 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.774998 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:41.775002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:41.775004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:41.775031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:41.776456 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.776895 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.776899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.776901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.776986 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:41.777265 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:41.777270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:41.777271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:41.777493 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:41.777718 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:41.777958 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 01:56:41.778075 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 01:56:41.780961 32048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:56:41.784015 32048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:56:41.799052 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.793208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.3.220.2:43365)
0809 01:56:41.793702 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:41.793717 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:41.793744 (+ 27us) server_negotiation.cc:407] Connection header received
0809 01:56:41.793831 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:41.793836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:41.794079 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:41.794249 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:41.794601 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.795530 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.797235 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.798175 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.798321 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.798449 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:41.798783 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 01:56:41.798883 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":145,"threads_started":1}
I20260809 01:56:41.799052 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.792492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:43365 (local address 127.0.0.1:48290)
0809 01:56:41.793209 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:41.793258 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:41.793412 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:41.794272 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:41.794279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:41.794306 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:41.794468 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.794475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.795663 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.795669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.797125 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.797136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.798505 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.798508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.798589 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.798617 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 01:56:41.798747 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 01:56:41.798883 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":46,"spinlock_wait_cycles":7424,"thread_start_us":118,"threads_started":1}
I20260809 01:56:41.799392 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.793097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.3.220.1:44917)
0809 01:56:41.793717 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:41.793731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:41.793895 (+ 164us) server_negotiation.cc:407] Connection header received
0809 01:56:41.793995 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:41.793999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:41.794148 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:41.794273 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:41.794730 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.795942 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.797452 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.798381 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.798486 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.798624 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:41.799122 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 01:56:41.799226 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":94,"threads_started":1}
I20260809 01:56:41.799706 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.792384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:45581 (local address 127.0.0.1:56460)
0809 01:56:41.793208 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:41.793257 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:41.793412 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:41.794771 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:41.794774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:41.794793 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:41.794887 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.794893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.796297 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.796302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.797788 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.797800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.799081 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.799085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.799397 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.799431 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 01:56:41.799509 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 01:56:41.799575 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":26,"spinlock_wait_cycles":3968,"thread_start_us":79,"threads_started":1}
I20260809 01:56:41.800155 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.792478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.3.220.3:45581)
0809 01:56:41.794308 (+ 1830us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:41.794322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:41.794345 (+ 23us) server_negotiation.cc:407] Connection header received
0809 01:56:41.794419 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:41.794426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:41.794628 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:41.794797 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:41.795009 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.796105 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.797922 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.798760 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.798871 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.799015 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:41.799958 (+ 943us) server_negotiation.cc:299] Negotiation successful
0809 01:56:41.800039 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714,"thread_start_us":137,"threads_started":1}
I20260809 01:56:41.799705 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:41.792813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:44917 (local address 127.0.0.1:35710)
0809 01:56:41.793786 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:41.793800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:41.793893 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:41.794457 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:41.794459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:41.794475 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:41.794595 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.794602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.796093 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.796096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.797338 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:41.797346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:41.798692 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:41.798696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:41.798782 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:41.798836 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 01:56:41.798901 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 01:56:41.798973 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":44,"thread_start_us":101,"threads_started":1}
I20260809 01:56:41.802799 32636 tablet_service.cc:1467] Processing CreateTablet for tablet 571c0c7d41084a319ebe78bcd02035a9 (DEFAULT_TABLE table=test-table [id=df78709191a24e44ba49763ddf99ae34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:41.802616 32422 tablet_service.cc:1467] Processing CreateTablet for tablet 571c0c7d41084a319ebe78bcd02035a9 (DEFAULT_TABLE table=test-table [id=df78709191a24e44ba49763ddf99ae34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:41.803071 32229 tablet_service.cc:1467] Processing CreateTablet for tablet 571c0c7d41084a319ebe78bcd02035a9 (DEFAULT_TABLE table=test-table [id=df78709191a24e44ba49763ddf99ae34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:41.803781 32636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571c0c7d41084a319ebe78bcd02035a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:41.803802 32422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571c0c7d41084a319ebe78bcd02035a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:41.803998 32229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571c0c7d41084a319ebe78bcd02035a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:41.810223 32762 tablet_bootstrap.cc:492] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Bootstrap starting.
I20260809 01:56:41.811513 32764 tablet_bootstrap.cc:492] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1: Bootstrap starting.
I20260809 01:56:41.811899 32765 tablet_bootstrap.cc:492] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Bootstrap starting.
I20260809 01:56:41.812842 32762 tablet_bootstrap.cc:654] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:41.814177 32765 tablet_bootstrap.cc:654] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:41.814129 32762 log.cc:824] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:41.815069 32765 log.cc:824] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:41.815840 32764 tablet_bootstrap.cc:654] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:41.816582 32762 tablet_bootstrap.cc:492] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: No bootstrap required, opened a new log
I20260809 01:56:41.816799 32762 ts_tablet_manager.cc:1397] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 01:56:41.817070 32764 log.cc:824] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:41.817224 32765 tablet_bootstrap.cc:492] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: No bootstrap required, opened a new log
I20260809 01:56:41.817417 32765 ts_tablet_manager.cc:1397] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:56:41.819072 32764 tablet_bootstrap.cc:492] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1: No bootstrap required, opened a new log
I20260809 01:56:41.819264 32764 ts_tablet_manager.cc:1397] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260809 01:56:41.820694 32762 raft_consensus.cc:348] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:41.820953 32762 raft_consensus.cc:374] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:41.821028 32762 raft_consensus.cc:729] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418b468ec74446a08bf6a14f64bcaba3, State: Initialized, Role: FOLLOWER
I20260809 01:56:41.821308 32765 raft_consensus.cc:348] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:41.821480 32762 consensus_queue.cc:260] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:41.821632 32765 raft_consensus.cc:374] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:41.821710 32765 raft_consensus.cc:729] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10f2d9fead64d6ab4e2a7cfcf289d93, State: Initialized, Role: FOLLOWER
I20260809 01:56:41.822186 32737 heartbeater.cc:503] Master 127.3.220.62:39977 was elected leader, sending a full tablet report...
I20260809 01:56:41.822211 32765 consensus_queue.cc:260] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:41.822551 32762 ts_tablet_manager.cc:1428] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 01:56:41.823160 32765 ts_tablet_manager.cc:1428] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260809 01:56:41.823027 32764 raft_consensus.cc:348] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:41.823231 32527 heartbeater.cc:503] Master 127.3.220.62:39977 was elected leader, sending a full tablet report...
I20260809 01:56:41.823307 32764 raft_consensus.cc:374] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:41.823360 32764 raft_consensus.cc:729] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206b3892efef4111b920815492b264a1, State: Initialized, Role: FOLLOWER
I20260809 01:56:41.823761 32764 consensus_queue.cc:260] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:41.824483 32320 heartbeater.cc:503] Master 127.3.220.62:39977 was elected leader, sending a full tablet report...
I20260809 01:56:41.824664 32764 ts_tablet_manager.cc:1428] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260809 01:56:41.992340 32738 tablet.cc:2369] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:56:42.018566 307 raft_consensus.cc:482] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:42.018832 307 raft_consensus.cc:504] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:42.020054 307 leader_election.cc:288] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418b468ec74446a08bf6a14f64bcaba3 (127.3.220.3:45581), a10f2d9fead64d6ab4e2a7cfcf289d93 (127.3.220.2:43365)
W20260809 01:56:42.020615 32529 tablet.cc:2369] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:56:42.024871 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.020982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:41275 (local address 127.3.220.3:45581)
0809 01:56:42.021107 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:42.021111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:42.021124 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:56:42.021181 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:42.021184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:42.021245 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:42.021344 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:42.021699 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.022488 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.023866 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.024280 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.024409 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.024472 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:42.024676 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 01:56:42.024751 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:56:42.024896 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.020340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:45581 (local address 127.3.220.1:41275)
0809 01:56:42.020693 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:42.020710 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:42.020856 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:42.021364 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:42.021367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:42.021388 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:42.021513 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:42.021518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.022620 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.022624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:42.023735 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:42.023742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.024426 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.024430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:42.024533 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.024558 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 01:56:42.024653 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 01:56:42.024731 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":164,"threads_started":1}
I20260809 01:56:42.024988 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.020467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:41951 (local address 127.3.220.2:43365)
0809 01:56:42.020823 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:42.020828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:42.020849 (+ 21us) server_negotiation.cc:407] Connection header received
0809 01:56:42.021053 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:42.021057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:42.021106 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:42.021173 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:42.021612 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.022162 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.023696 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.024284 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.024389 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.024454 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:42.024771 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 01:56:42.024864 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260809 01:56:42.025388 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.020353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:43365 (local address 127.3.220.1:41951)
0809 01:56:42.020783 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:42.020796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:42.020866 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:42.021299 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:42.021302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:42.021319 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:42.021464 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:42.021471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.022300 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.022306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:42.023558 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:42.023569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.024514 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.024518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:42.024639 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.024661 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 01:56:42.025168 (+ 507us) client_negotiation.cc:240] Negotiation successful
0809 01:56:42.025243 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260809 01:56:42.025736 32667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571c0c7d41084a319ebe78bcd02035a9" candidate_uuid: "206b3892efef4111b920815492b264a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418b468ec74446a08bf6a14f64bcaba3" is_pre_election: true
I20260809 01:56:42.026039 32452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571c0c7d41084a319ebe78bcd02035a9" candidate_uuid: "206b3892efef4111b920815492b264a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" is_pre_election: true
I20260809 01:56:42.026096 32667 raft_consensus.cc:2454] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 206b3892efef4111b920815492b264a1 in term 0.
I20260809 01:56:42.026314 32452 raft_consensus.cc:2454] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 206b3892efef4111b920815492b264a1 in term 0.
I20260809 01:56:42.026656 32164 leader_election.cc:302] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206b3892efef4111b920815492b264a1, 418b468ec74446a08bf6a14f64bcaba3; no voters:
I20260809 01:56:42.026975 307 raft_consensus.cc:2781] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:56:42.027076 307 raft_consensus.cc:482] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:56:42.027117 307 raft_consensus.cc:3037] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:42.028431 307 raft_consensus.cc:504] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:42.028803 307 leader_election.cc:288] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [CANDIDATE]: Term 1 election: Requested vote from peers 418b468ec74446a08bf6a14f64bcaba3 (127.3.220.3:45581), a10f2d9fead64d6ab4e2a7cfcf289d93 (127.3.220.2:43365)
I20260809 01:56:42.029277 32452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571c0c7d41084a319ebe78bcd02035a9" candidate_uuid: "206b3892efef4111b920815492b264a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93"
I20260809 01:56:42.029277 32667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571c0c7d41084a319ebe78bcd02035a9" candidate_uuid: "206b3892efef4111b920815492b264a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418b468ec74446a08bf6a14f64bcaba3"
I20260809 01:56:42.029433 32452 raft_consensus.cc:3037] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:42.029433 32667 raft_consensus.cc:3037] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:42.030845 32452 raft_consensus.cc:2454] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 206b3892efef4111b920815492b264a1 in term 1.
I20260809 01:56:42.030838 32667 raft_consensus.cc:2454] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 206b3892efef4111b920815492b264a1 in term 1.
I20260809 01:56:42.031169 32164 leader_election.cc:302] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206b3892efef4111b920815492b264a1, 418b468ec74446a08bf6a14f64bcaba3; no voters:
I20260809 01:56:42.031387 307 raft_consensus.cc:2781] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:42.031647 307 raft_consensus.cc:686] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 1 LEADER]: Becoming Leader. State: Replica: 206b3892efef4111b920815492b264a1, State: Running, Role: LEADER
I20260809 01:56:42.032019 307 consensus_queue.cc:237] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } }
I20260809 01:56:42.035419 32047 catalog_manager.cc:5495] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 206b3892efef4111b920815492b264a1 (127.3.220.1). New cstate: current_term: 1 leader_uuid: "206b3892efef4111b920815492b264a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "206b3892efef4111b920815492b264a1" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 44917 } health_report { overall_health: HEALTHY } } }
W20260809 01:56:42.055974 32321 tablet.cc:2369] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:56:42.097259 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.093925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.3.220.1:44917)
0809 01:56:42.094061 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:42.094065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:42.094080 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:56:42.094188 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:42.094191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:42.094249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:42.094333 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:56:42.094567 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.095074 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.095884 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.096089 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.096164 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.096429 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:56:42.096882 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:56:42.096987 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:42.097042 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 01:56:42.097093 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 01:56:42.103638 32452 raft_consensus.cc:1261] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 1 FOLLOWER]: Refusing update from remote peer 206b3892efef4111b920815492b264a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:42.103638 32667 raft_consensus.cc:1261] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 1 FOLLOWER]: Refusing update from remote peer 206b3892efef4111b920815492b264a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:42.104513 310 consensus_queue.cc:1035] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10f2d9fead64d6ab4e2a7cfcf289d93" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:42.104730 307 consensus_queue.cc:1035] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "418b468ec74446a08bf6a14f64bcaba3" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 45581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:42.122985 32048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:42.123196 32048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:42.125702 32048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdf60606945641ab9560adbbbf9a787f: Sending DeleteTablet for 3 replicas of tablet 571c0c7d41084a319ebe78bcd02035a9
I20260809 01:56:42.126507 32636 tablet_service.cc:1514] Processing DeleteTablet for tablet 571c0c7d41084a319ebe78bcd02035a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:42 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:56460
I20260809 01:56:42.126514 32229 tablet_service.cc:1514] Processing DeleteTablet for tablet 571c0c7d41084a319ebe78bcd02035a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:42 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:35710
I20260809 01:56:42.126571 32422 tablet_service.cc:1514] Processing DeleteTablet for tablet 571c0c7d41084a319ebe78bcd02035a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:42 UTC) from {username='kudu', principal='kudu/127.3.220.62@KRBTEST.COM'} at 127.0.0.1:48290
I20260809 01:56:42.127038 329 tablet_replica.cc:330] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1: stopping tablet replica
I20260809 01:56:42.127056 330 tablet_replica.cc:330] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: stopping tablet replica
I20260809 01:56:42.127374 330 raft_consensus.cc:2229] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:42.127390 329 raft_consensus.cc:2229] T 571c0c7d41084a319ebe78bcd02035a9 P 206b3892efef4111b920815492b264a1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:56:42.127629 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 32140
I20260809 01:56:42.127758 330 raft_consensus.cc:2258] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:42.130301 331 tablet_replica.cc:330] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: stopping tablet replica
I20260809 01:56:42.130584 331 raft_consensus.cc:2229] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:42.130827 331 raft_consensus.cc:2258] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:42.131832 330 ts_tablet_manager.cc:1905] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:42.133654 331 ts_tablet_manager.cc:1905] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:42.135191 330 ts_tablet_manager.cc:1918] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:42.135315 330 log.cc:1196] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/571c0c7d41084a319ebe78bcd02035a9
I20260809 01:56:42.135628 330 ts_tablet_manager.cc:1939] T 571c0c7d41084a319ebe78bcd02035a9 P 418b468ec74446a08bf6a14f64bcaba3: Deleting consensus metadata
I20260809 01:56:42.137189 331 ts_tablet_manager.cc:1918] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:42.137157 32029 catalog_manager.cc:4841] TS 418b468ec74446a08bf6a14f64bcaba3 (127.3.220.3:45581): tablet 571c0c7d41084a319ebe78bcd02035a9 (table test-table [id=df78709191a24e44ba49763ddf99ae34]) successfully deleted
I20260809 01:56:42.137331 331 log.cc:1196] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/571c0c7d41084a319ebe78bcd02035a9
I20260809 01:56:42.137692 331 ts_tablet_manager.cc:1939] T 571c0c7d41084a319ebe78bcd02035a9 P a10f2d9fead64d6ab4e2a7cfcf289d93: Deleting consensus metadata
I20260809 01:56:42.139096 32030 catalog_manager.cc:4841] TS a10f2d9fead64d6ab4e2a7cfcf289d93 (127.3.220.2:43365): tablet 571c0c7d41084a319ebe78bcd02035a9 (table test-table [id=df78709191a24e44ba49763ddf99ae34]) successfully deleted
W20260809 01:56:42.143021 32027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:44917 (error 108)
I20260809 01:56:42.143994 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 32330
I20260809 01:56:42.144224 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.143865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:44917 (local address 127.0.0.1:35726)
0809 01:56:42.144022 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:42.144111 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 01:56:42.149901 32027 catalog_manager.cc:4584] TS 206b3892efef4111b920815492b264a1 (127.3.220.1:44917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 571c0c7d41084a319ebe78bcd02035a9: Network error: Client connection negotiation failed: client connection to 127.3.220.1:44917: connect: Connection refused (error 111)
I20260809 01:56:42.159384 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 32535
I20260809 01:56:42.177027 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 32005
2026-08-09T01:56:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2736 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T01:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:42Z Disabled control of system clock
I20260809 01:56:42.216127 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:40149
--webserver_interface=127.3.220.62
--webserver_port=0
--builtin_ntp_servers=127.3.220.20:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 01:56:42.329861 338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:42.330137 338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:42.330171 338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:42.334024 338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:42.334075 338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:42.334090 338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:42.334112 338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:42.334127 338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:42.338891 338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:40149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:42.339872 338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:42.341210 338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:42.349519 350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:42.349596 345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:42.349581 347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:42.349849 338 server_base.cc:1034] running on GCE node
I20260809 01:56:42.350425 338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:42.351509 338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:42.352798 338 hybrid_clock.cc:648] HybridClock initialized: now 1786240602352780 us; error 41 us; skew 500 ppm
I20260809 01:56:42.358733 338 webserver.cc:460] Webserver started at http://127.3.220.62:43917/ using document root <none> and password file <none>
I20260809 01:56:42.359347 338 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:42.359442 338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:42.359684 338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:42.361857 338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "a34a401a4fc8418e93432a8818e816ea"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "e2625d24c3838b140f359c7e39a3e067"
server_key_iv: "55ea4fc3427a32938839bd4667dc261e"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.362437 338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "a34a401a4fc8418e93432a8818e816ea"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "e2625d24c3838b140f359c7e39a3e067"
server_key_iv: "55ea4fc3427a32938839bd4667dc261e"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.366103 338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 01:56:42.368422 358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:42.369704 338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:42.369843 338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "a34a401a4fc8418e93432a8818e816ea"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "e2625d24c3838b140f359c7e39a3e067"
server_key_iv: "55ea4fc3427a32938839bd4667dc261e"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.369953 338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:42.398655 338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:42.399422 338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:42.399637 338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:42.407132 440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:40149 every 8 connection(s)
I20260809 01:56:42.407140 338 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:40149
I20260809 01:56:42.408145 338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:42.411008 443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:42.412403 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 338
I20260809 01:56:42.412530 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:42.412868 3952 external_mini_cluster.cc:1428] Setting key c848770ee9a9a13e251fb6541389ca4d
I20260809 01:56:42.417459 443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea: Bootstrap starting.
I20260809 01:56:42.420256 443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:42.421057 443 log.cc:824] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:42.422344 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.414366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.3.220.62:40149)
0809 01:56:42.414982 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:42.414998 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:42.415034 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:56:42.415666 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:42.415683 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:42.416076 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:42.416350 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:42.416757 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.417789 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.419039 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.419502 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.419614 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.420501 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:42.420506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:42.420519 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:42.420916 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:42.420973 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:42.420993 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:42.421069 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:42.421272 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 01:56:42.421602 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":130,"threads_started":1}
I20260809 01:56:42.423139 443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea: No bootstrap required, opened a new log
I20260809 01:56:42.427042 443 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34a401a4fc8418e93432a8818e816ea" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40149 } }
I20260809 01:56:42.427289 443 raft_consensus.cc:374] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:42.427371 443 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34a401a4fc8418e93432a8818e816ea, State: Initialized, Role: FOLLOWER
I20260809 01:56:42.427795 443 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34a401a4fc8418e93432a8818e816ea" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40149 } }
I20260809 01:56:42.427946 443 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:42.428033 443 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:42.428153 443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:42.429139 443 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34a401a4fc8418e93432a8818e816ea" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40149 } }
I20260809 01:56:42.429481 443 leader_election.cc:302] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34a401a4fc8418e93432a8818e816ea; no voters:
I20260809 01:56:42.429896 443 leader_election.cc:288] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:42.429993 448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:42.430254 448 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [term 1 LEADER]: Becoming Leader. State: Replica: a34a401a4fc8418e93432a8818e816ea, State: Running, Role: LEADER
I20260809 01:56:42.430675 448 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34a401a4fc8418e93432a8818e816ea" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40149 } }
I20260809 01:56:42.431198 443 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:42.431838 449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34a401a4fc8418e93432a8818e816ea. Latest consensus state: current_term: 1 leader_uuid: "a34a401a4fc8418e93432a8818e816ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34a401a4fc8418e93432a8818e816ea" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40149 } } }
I20260809 01:56:42.431967 449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:42.432359 450 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a34a401a4fc8418e93432a8818e816ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34a401a4fc8418e93432a8818e816ea" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40149 } } }
I20260809 01:56:42.432469 450 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:42.432731 455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:42.435819 455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:42.441015 455 catalog_manager.cc:1347] Generated new cluster ID: 684a77487d504694a271a9963e444d87
I20260809 01:56:42.441093 455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:42.470057 455 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:42.471231 455 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:42.479849 455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea: Generated new TSK 0
I20260809 01:56:42.480583 455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:42.490645 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.1:0
--local_ip_for_outbound_sockets=127.3.220.1
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:40149
--builtin_ntp_servers=127.3.220.20:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 01:56:42.608150 478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:42.608417 478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:42.608456 478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:42.612529 478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:42.612587 478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:42.612643 478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.1
I20260809 01:56:42.617673 478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.220.1
--webserver_port=0
--tserver_master_addrs=127.3.220.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.1
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:42.618817 478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:42.620530 478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:42.631304 484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:42.631304 486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:42.631338 490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:42.631503 478 server_base.cc:1034] running on GCE node
I20260809 01:56:42.632146 478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:42.632957 478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:42.634274 478 hybrid_clock.cc:648] HybridClock initialized: now 1786240602634253 us; error 39 us; skew 500 ppm
I20260809 01:56:42.636246 478 webserver.cc:460] Webserver started at http://127.3.220.1:34883/ using document root <none> and password file <none>
I20260809 01:56:42.636870 478 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:42.636973 478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:42.637212 478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:42.639447 478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/data/instance:
uuid: "debefd480d4748db8d270fd0c53b2d32"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "559022b9ad625ab9a45e6c3acc49eb6d"
server_key_iv: "181cab844dedae7746af443cd60e7472"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.640026 478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance:
uuid: "debefd480d4748db8d270fd0c53b2d32"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "559022b9ad625ab9a45e6c3acc49eb6d"
server_key_iv: "181cab844dedae7746af443cd60e7472"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.643841 478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260809 01:56:42.646319 499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:42.647584 478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:56:42.647722 478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/wal
uuid: "debefd480d4748db8d270fd0c53b2d32"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "559022b9ad625ab9a45e6c3acc49eb6d"
server_key_iv: "181cab844dedae7746af443cd60e7472"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.647817 478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:42.663483 478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:42.664260 478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:42.664482 478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:42.665127 478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:42.666188 478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:42.666236 478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:42.666308 478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:42.666347 478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:42.676303 478 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.1:38339
I20260809 01:56:42.676352 662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.1:38339 every 8 connection(s)
I20260809 01:56:42.677479 478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/data/info.pb
I20260809 01:56:42.678272 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 478
I20260809 01:56:42.678432 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-0/wal/instance
I20260809 01:56:42.678726 3952 external_mini_cluster.cc:1428] Setting key 7fba0893874870938e744610e663c147
I20260809 01:56:42.685158 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.2:0
--local_ip_for_outbound_sockets=127.3.220.2
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:40149
--builtin_ntp_servers=127.3.220.20:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 01:56:42.689610 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.679562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.1:35495 (local address 127.3.220.62:40149)
0809 01:56:42.684218 (+ 4656us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:42.684222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:42.684236 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:56:42.684275 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:42.684277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:42.684355 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:42.684443 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:42.685632 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.686199 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.687488 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.687726 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.687851 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.689025 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:42.689031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:42.689039 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:42.689052 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:42.689072 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:42.689079 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:42.689160 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:42.689335 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 01:56:42.689410 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4563}
I20260809 01:56:42.690579 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.680131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:40149 (local address 127.3.220.1:35495)
0809 01:56:42.680931 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:42.680987 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:42.682570 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:42.684683 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:42.684693 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:42.684812 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:42.685447 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:42.685464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.686353 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.686357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:42.687343 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:42.687353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.687899 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.687903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:42.688086 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.688747 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:42.688749 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:42.688792 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:42.688861 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:42.689162 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:42.689176 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:42.689183 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 01:56:42.689335 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 01:56:42.689689 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260809 01:56:42.691936 664 heartbeater.cc:347] Connected to a master server at 127.3.220.62:40149
I20260809 01:56:42.692206 664 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:42.692766 664 heartbeater.cc:511] Master 127.3.220.62:40149 requested a full tablet report, sending...
I20260809 01:56:42.694615 383 ts_manager.cc:194] Registered new tserver with Master: debefd480d4748db8d270fd0c53b2d32 (127.3.220.1:38339)
I20260809 01:56:42.696465 383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.1:35495
W20260809 01:56:42.801491 668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:42.801783 668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:42.801822 668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:42.806627 668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:42.806689 668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:42.806775 668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.2
I20260809 01:56:42.811630 668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.220.2
--webserver_port=0
--tserver_master_addrs=127.3.220.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.2
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:42.812795 668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:42.814317 668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:42.824550 676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:42.824543 681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:42.824543 678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:42.824771 668 server_base.cc:1034] running on GCE node
I20260809 01:56:42.825196 668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:42.825917 668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:42.827116 668 hybrid_clock.cc:648] HybridClock initialized: now 1786240602827085 us; error 51 us; skew 500 ppm
I20260809 01:56:42.829016 668 webserver.cc:460] Webserver started at http://127.3.220.2:37401/ using document root <none> and password file <none>
I20260809 01:56:42.829653 668 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:42.829725 668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:42.829919 668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:42.832165 668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/data/instance:
uuid: "54899f0cfe744fafa3ed56d78caea360"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "09e46ff58501491ba78a8aed9f23c0a0"
server_key_iv: "b193f591555ca9cc4926701c0e78af99"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.832794 668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance:
uuid: "54899f0cfe744fafa3ed56d78caea360"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "09e46ff58501491ba78a8aed9f23c0a0"
server_key_iv: "b193f591555ca9cc4926701c0e78af99"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.837215 668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260809 01:56:42.839891 688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:42.841229 668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:56:42.841382 668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/wal
uuid: "54899f0cfe744fafa3ed56d78caea360"
format_stamp: "Formatted at 2026-08-09 01:56:42 on dist-test-slave-gkw7"
server_key: "09e46ff58501491ba78a8aed9f23c0a0"
server_key_iv: "b193f591555ca9cc4926701c0e78af99"
server_key_version: "encryptionkey@0"
I20260809 01:56:42.841516 668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:42.856763 668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:42.857656 668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:42.857882 668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:42.858534 668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:42.859530 668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:42.859606 668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:42.859675 668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:42.859728 668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:42.868631 668 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.2:46873
I20260809 01:56:42.868703 853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.2:46873 every 8 connection(s)
I20260809 01:56:42.869743 668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/data/info.pb
I20260809 01:56:42.873261 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 668
I20260809 01:56:42.873389 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/wal/instance
I20260809 01:56:42.873737 3952 external_mini_cluster.cc:1428] Setting key 23ce45dfaf2b63318da0a0c7b509ea8a
I20260809 01:56:42.876065 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.220.3:0
--local_ip_for_outbound_sockets=127.3.220.3
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:40149
--builtin_ntp_servers=127.3.220.20:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 01:56:42.880666 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.871651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.2:43879 (local address 127.3.220.62:40149)
0809 01:56:42.871803 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:42.871808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:42.872939 (+ 1131us) server_negotiation.cc:407] Connection header received
0809 01:56:42.874850 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:42.874853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:42.874899 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:42.874986 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:42.876214 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.876769 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.878446 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.878838 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.878961 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.880057 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:42.880063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:42.880070 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:42.880089 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:42.880117 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:42.880130 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:42.880214 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:42.880359 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 01:56:42.880437 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:56:42.881422 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:42.872134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:40149 (local address 127.3.220.2:43879)
0809 01:56:42.872802 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:42.872846 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:42.874647 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:42.875188 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:42.875197 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:42.875306 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:42.876032 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:42.876056 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.876911 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.876918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:42.878310 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:42.878327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:42.878998 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:42.879001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:42.879137 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:42.879779 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:42.879781 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:42.879823 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:42.879888 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:42.880220 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:42.880227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:42.880231 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 01:56:42.880334 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 01:56:42.880555 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260809 01:56:42.882704 854 heartbeater.cc:347] Connected to a master server at 127.3.220.62:40149
I20260809 01:56:42.882937 854 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:42.883477 854 heartbeater.cc:511] Master 127.3.220.62:40149 requested a full tablet report, sending...
I20260809 01:56:42.884580 383 ts_manager.cc:194] Registered new tserver with Master: 54899f0cfe744fafa3ed56d78caea360 (127.3.220.2:46873)
I20260809 01:56:42.885213 383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.2:43879
W20260809 01:56:42.992359 861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:42.992645 861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:42.992686 861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:42.996747 861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:42.996824 861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:42.996881 861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.220.3
I20260809 01:56:43.001753 861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.220.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.220.3
--webserver_port=0
--tserver_master_addrs=127.3.220.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.220.3
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:43.002954 861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:43.004398 861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:43.014041 867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:43.014024 868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:43.014024 870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:43.014276 861 server_base.cc:1034] running on GCE node
I20260809 01:56:43.014813 861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:43.015487 861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:43.016683 861 hybrid_clock.cc:648] HybridClock initialized: now 1786240603016662 us; error 31 us; skew 500 ppm
I20260809 01:56:43.018607 861 webserver.cc:460] Webserver started at http://127.3.220.3:35275/ using document root <none> and password file <none>
I20260809 01:56:43.019241 861 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:43.019340 861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:43.019575 861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:43.021749 861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/data/instance:
uuid: "a687d815ce1344cc85a2753c14c09d5c"
format_stamp: "Formatted at 2026-08-09 01:56:43 on dist-test-slave-gkw7"
server_key: "6f595f7fe6136830aa0f527efac6386b"
server_key_iv: "2dd870ded6dda0433318a0e4ea0d9880"
server_key_version: "encryptionkey@0"
I20260809 01:56:43.022362 861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance:
uuid: "a687d815ce1344cc85a2753c14c09d5c"
format_stamp: "Formatted at 2026-08-09 01:56:43 on dist-test-slave-gkw7"
server_key: "6f595f7fe6136830aa0f527efac6386b"
server_key_iv: "2dd870ded6dda0433318a0e4ea0d9880"
server_key_version: "encryptionkey@0"
I20260809 01:56:43.026219 861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 01:56:43.028745 879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:43.030129 861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:43.030272 861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/wal
uuid: "a687d815ce1344cc85a2753c14c09d5c"
format_stamp: "Formatted at 2026-08-09 01:56:43 on dist-test-slave-gkw7"
server_key: "6f595f7fe6136830aa0f527efac6386b"
server_key_iv: "2dd870ded6dda0433318a0e4ea0d9880"
server_key_version: "encryptionkey@0"
I20260809 01:56:43.030396 861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:43.043426 861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:43.044112 861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:43.044319 861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:43.045001 861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 01:56:43.046062 861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 01:56:43.046136 861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:43.046211 861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 01:56:43.046253 861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:43.055804 861 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.3:35877
I20260809 01:56:43.055830 1055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.3:35877 every 8 connection(s)
I20260809 01:56:43.056784 861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/data/info.pb
I20260809 01:56:43.064254 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 861
I20260809 01:56:43.064405 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/wal/instance
I20260809 01:56:43.064680 3952 external_mini_cluster.cc:1428] Setting key 45737555cc39421a80257854d0ec1241
I20260809 01:56:43.066443 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.058914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:41755 (local address 127.3.220.62:40149)
0809 01:56:43.059062 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:43.059066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:43.060148 (+ 1082us) server_negotiation.cc:407] Connection header received
0809 01:56:43.061520 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:43.061523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:43.061598 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:43.061676 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:43.062572 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.063111 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.064293 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.064603 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.064725 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.065915 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:43.065918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:43.065921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:43.065935 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:43.065959 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:43.065973 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:43.066052 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:43.066234 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 01:56:43.066302 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 01:56:43.067437 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.059326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:40149 (local address 127.3.220.3:41755)
0809 01:56:43.059984 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:43.060025 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:43.061322 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:43.061823 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:43.061829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:43.061930 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:56:43.062433 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.062445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.063223 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.063225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.064148 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.064158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.064771 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.064775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.064957 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.065642 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 01:56:43.065644 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:56:43.065686 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 01:56:43.065751 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:56:43.066068 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:56:43.066081 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:56:43.066095 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 01:56:43.066227 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 01:56:43.066531 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
I20260809 01:56:43.068521 1056 heartbeater.cc:347] Connected to a master server at 127.3.220.62:40149
I20260809 01:56:43.068739 1056 heartbeater.cc:464] Registering TS with master...
I20260809 01:56:43.069206 1056 heartbeater.cc:511] Master 127.3.220.62:40149 requested a full tablet report, sending...
I20260809 01:56:43.070211 383 ts_manager.cc:194] Registered new tserver with Master: a687d815ce1344cc85a2753c14c09d5c (127.3.220.3:35877)
I20260809 01:56:43.070824 383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.220.3:41755
I20260809 01:56:43.078917 3952 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 01:56:43.084697 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.080959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.3.220.62:40149)
0809 01:56:43.081097 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:43.081100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:43.081316 (+ 216us) server_negotiation.cc:407] Connection header received
0809 01:56:43.081419 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:43.081421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:43.081462 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:43.081583 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:43.081895 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.082406 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.083252 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.083471 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.083550 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.084314 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:43.084317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:43.084327 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:43.084339 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:43.084359 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:43.084367 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:43.084424 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:43.084531 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 01:56:43.084584 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:56:43.087594 383 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 01:56:43.090670 383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 01:56:43.105579 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.100664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.3.220.2:46873)
0809 01:56:43.101142 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:43.101160 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:43.101186 (+ 26us) server_negotiation.cc:407] Connection header received
0809 01:56:43.101264 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:43.101268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:43.101455 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:43.101673 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:43.101874 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.102958 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.104316 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.104852 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.104979 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.105179 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:43.105287 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 01:56:43.105397 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":122,"threads_started":1}
I20260809 01:56:43.105571 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.099529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:38339 (local address 127.0.0.1:51964)
0809 01:56:43.100503 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:43.100518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:43.100602 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:43.101366 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:43.101369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:43.101385 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:43.101465 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.101469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.102684 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.102690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.104057 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.104069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.105043 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.105047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.105142 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.105164 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 01:56:43.105219 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 01:56:43.105352 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":63,"thread_start_us":102,"threads_started":1}
I20260809 01:56:43.105918 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.099581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.3.220.3:35877)
0809 01:56:43.100057 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:43.100071 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:43.100101 (+ 30us) server_negotiation.cc:407] Connection header received
0809 01:56:43.100314 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:43.100321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:43.100546 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:43.100707 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:43.101247 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.102524 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.104038 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.104879 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.105002 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.105191 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:43.105643 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 01:56:43.105750 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":160,"threads_started":1}
I20260809 01:56:43.105998 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.099439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.3:35877 (local address 127.0.0.1:49458)
0809 01:56:43.099914 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:43.099962 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:43.100088 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:43.100909 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:43.100916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:43.100943 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:43.101103 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.101111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.102655 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.102658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.103896 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.103909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.105253 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.105256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.105350 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.105370 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 01:56:43.105440 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 01:56:43.105502 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":102,"threads_started":1}
I20260809 01:56:43.106056 1077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.100280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.3.220.1:38339)
0809 01:56:43.100876 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:43.100895 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:43.100922 (+ 27us) server_negotiation.cc:407] Connection header received
0809 01:56:43.101004 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:43.101008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:43.101201 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:43.101663 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:43.101812 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.102565 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.104189 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.104910 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.105466 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.105688 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:43.105733 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 01:56:43.105832 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":121,"threads_started":1}
I20260809 01:56:43.106300 1072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.100183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:46873 (local address 127.0.0.1:46484)
0809 01:56:43.100750 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:43.100763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:43.100839 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:43.101639 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:43.101642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:43.101659 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:43.101751 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.101757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.103087 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.103091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.104199 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.104214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.104983 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.104987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.105090 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.105128 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 01:56:43.105259 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 01:56:43.105379 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":25,"thread_start_us":99,"threads_started":1}
I20260809 01:56:43.109231 569 tablet_service.cc:1467] Processing CreateTablet for tablet c6491c681ff646dc8cd096b481215114 (DEFAULT_TABLE table=test-table [id=68ef8c0c08e54c9f8831e59e95d2c301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:43.109231 949 tablet_service.cc:1467] Processing CreateTablet for tablet c6491c681ff646dc8cd096b481215114 (DEFAULT_TABLE table=test-table [id=68ef8c0c08e54c9f8831e59e95d2c301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:43.110476 949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6491c681ff646dc8cd096b481215114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:43.110476 569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6491c681ff646dc8cd096b481215114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:43.112188 760 tablet_service.cc:1467] Processing CreateTablet for tablet c6491c681ff646dc8cd096b481215114 (DEFAULT_TABLE table=test-table [id=68ef8c0c08e54c9f8831e59e95d2c301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 01:56:43.113071 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6491c681ff646dc8cd096b481215114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:43.118163 1079 tablet_bootstrap.cc:492] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32: Bootstrap starting.
I20260809 01:56:43.118631 1080 tablet_bootstrap.cc:492] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Bootstrap starting.
I20260809 01:56:43.119490 1083 tablet_bootstrap.cc:492] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Bootstrap starting.
I20260809 01:56:43.121063 1080 tablet_bootstrap.cc:654] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:43.121456 1083 tablet_bootstrap.cc:654] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:43.122061 1080 log.cc:824] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:43.122310 1083 log.cc:824] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:43.124161 1083 tablet_bootstrap.cc:492] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: No bootstrap required, opened a new log
I20260809 01:56:43.124159 1080 tablet_bootstrap.cc:492] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: No bootstrap required, opened a new log
I20260809 01:56:43.124275 1079 tablet_bootstrap.cc:654] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:43.124357 1080 ts_tablet_manager.cc:1397] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 01:56:43.124357 1083 ts_tablet_manager.cc:1397] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 01:56:43.124955 1079 log.cc:824] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:43.127120 1079 tablet_bootstrap.cc:492] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32: No bootstrap required, opened a new log
I20260809 01:56:43.127339 1079 ts_tablet_manager.cc:1397] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260809 01:56:43.128862 1080 raft_consensus.cc:348] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.128865 1083 raft_consensus.cc:348] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.129145 1083 raft_consensus.cc:374] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:43.129145 1080 raft_consensus.cc:374] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:43.129208 1080 raft_consensus.cc:729] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a687d815ce1344cc85a2753c14c09d5c, State: Initialized, Role: FOLLOWER
I20260809 01:56:43.129237 1083 raft_consensus.cc:729] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54899f0cfe744fafa3ed56d78caea360, State: Initialized, Role: FOLLOWER
I20260809 01:56:43.129755 1080 consensus_queue.cc:260] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.129755 1083 consensus_queue.cc:260] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.130484 854 heartbeater.cc:503] Master 127.3.220.62:40149 was elected leader, sending a full tablet report...
I20260809 01:56:43.130709 1056 heartbeater.cc:503] Master 127.3.220.62:40149 was elected leader, sending a full tablet report...
I20260809 01:56:43.130785 1083 ts_tablet_manager.cc:1428] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 01:56:43.131189 1079 raft_consensus.cc:348] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.131431 1079 raft_consensus.cc:374] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:43.131518 1079 raft_consensus.cc:729] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debefd480d4748db8d270fd0c53b2d32, State: Initialized, Role: FOLLOWER
I20260809 01:56:43.132124 1080 ts_tablet_manager.cc:1428] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260809 01:56:43.132229 1079 consensus_queue.cc:260] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.132995 664 heartbeater.cc:503] Master 127.3.220.62:40149 was elected leader, sending a full tablet report...
I20260809 01:56:43.133244 1079 ts_tablet_manager.cc:1428] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260809 01:56:43.178866 666 tablet.cc:2369] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 01:56:43.307911 1057 tablet.cc:2369] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 01:56:43.370991 855 tablet.cc:2369] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:56:43.428929 1090 raft_consensus.cc:482] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 01:56:43.429164 1090 raft_consensus.cc:504] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.430506 1090 leader_election.cc:288] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers debefd480d4748db8d270fd0c53b2d32 (127.3.220.1:38339), 54899f0cfe744fafa3ed56d78caea360 (127.3.220.2:46873)
I20260809 01:56:43.434955 1077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.430965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:56589 (local address 127.3.220.1:38339)
0809 01:56:43.431103 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:43.431108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:43.431123 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:56:43.431185 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:43.431191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:43.431240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:43.431341 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:43.431746 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.432498 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.433782 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.434335 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.434475 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.434567 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:43.434724 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 01:56:43.434823 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:56:43.434930 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.430717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:38339 (local address 127.3.220.3:56589)
0809 01:56:43.430848 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:43.430864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:43.430976 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:43.431382 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:43.431389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:43.431443 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:43.431623 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.431632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.432610 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.432614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.433657 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.433667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.434482 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.434488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.434594 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.434621 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 01:56:43.434700 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 01:56:43.434755 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":29,"spinlock_wait_cycles":27264}
I20260809 01:56:43.435056 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.430942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.220.3:60501 (local address 127.3.220.2:46873)
0809 01:56:43.431072 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:43.431076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:43.431400 (+ 324us) server_negotiation.cc:407] Connection header received
0809 01:56:43.431565 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:43.431570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:43.431626 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:43.431713 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 01:56:43.432016 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.432660 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.434019 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.434462 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.434571 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.434632 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:43.434858 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 01:56:43.434926 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:56:43.435091 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.430838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.2:46873 (local address 127.3.220.3:60501)
0809 01:56:43.431307 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:43.431322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 01:56:43.431453 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:56:43.431742 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:56:43.431745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:56:43.431764 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 01:56:43.431893 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.431899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.432789 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.432792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.433866 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:56:43.433877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.434589 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.434593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:56:43.434718 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.434742 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 01:56:43.434818 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 01:56:43.434881 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":53,"spinlock_wait_cycles":7936,"thread_start_us":146,"threads_started":1}
I20260809 01:56:43.435685 787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6491c681ff646dc8cd096b481215114" candidate_uuid: "a687d815ce1344cc85a2753c14c09d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54899f0cfe744fafa3ed56d78caea360" is_pre_election: true
I20260809 01:56:43.435688 595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6491c681ff646dc8cd096b481215114" candidate_uuid: "a687d815ce1344cc85a2753c14c09d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "debefd480d4748db8d270fd0c53b2d32" is_pre_election: true
I20260809 01:56:43.435995 595 raft_consensus.cc:2454] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a687d815ce1344cc85a2753c14c09d5c in term 0.
I20260809 01:56:43.436007 787 raft_consensus.cc:2454] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a687d815ce1344cc85a2753c14c09d5c in term 0.
I20260809 01:56:43.436534 884 leader_election.cc:302] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 54899f0cfe744fafa3ed56d78caea360, a687d815ce1344cc85a2753c14c09d5c, debefd480d4748db8d270fd0c53b2d32; no voters:
I20260809 01:56:43.436842 1090 raft_consensus.cc:2781] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 01:56:43.436942 1090 raft_consensus.cc:482] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 01:56:43.437002 1090 raft_consensus.cc:3037] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:43.438192 1090 raft_consensus.cc:504] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.438598 1090 leader_election.cc:288] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [CANDIDATE]: Term 1 election: Requested vote from peers debefd480d4748db8d270fd0c53b2d32 (127.3.220.1:38339), 54899f0cfe744fafa3ed56d78caea360 (127.3.220.2:46873)
I20260809 01:56:43.438966 595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6491c681ff646dc8cd096b481215114" candidate_uuid: "a687d815ce1344cc85a2753c14c09d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "debefd480d4748db8d270fd0c53b2d32"
I20260809 01:56:43.438989 787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6491c681ff646dc8cd096b481215114" candidate_uuid: "a687d815ce1344cc85a2753c14c09d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54899f0cfe744fafa3ed56d78caea360"
I20260809 01:56:43.439078 595 raft_consensus.cc:3037] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:43.439136 787 raft_consensus.cc:3037] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:43.440163 595 raft_consensus.cc:2454] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a687d815ce1344cc85a2753c14c09d5c in term 1.
I20260809 01:56:43.440184 787 raft_consensus.cc:2454] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a687d815ce1344cc85a2753c14c09d5c in term 1.
I20260809 01:56:43.440521 884 leader_election.cc:302] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a687d815ce1344cc85a2753c14c09d5c, debefd480d4748db8d270fd0c53b2d32; no voters:
I20260809 01:56:43.440728 1090 raft_consensus.cc:2781] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:43.440961 1090 raft_consensus.cc:686] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 1 LEADER]: Becoming Leader. State: Replica: a687d815ce1344cc85a2753c14c09d5c, State: Running, Role: LEADER
I20260809 01:56:43.441351 1090 consensus_queue.cc:237] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } }
I20260809 01:56:43.444809 383 catalog_manager.cc:5495] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to a687d815ce1344cc85a2753c14c09d5c (127.3.220.3). New cstate: current_term: 1 leader_uuid: "a687d815ce1344cc85a2753c14c09d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a687d815ce1344cc85a2753c14c09d5c" member_type: VOTER last_known_addr { host: "127.3.220.3" port: 35877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 } health_report { overall_health: UNKNOWN } } }
I20260809 01:56:43.468053 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.464795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.3.220.3:35877)
0809 01:56:43.464938 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:43.464941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:43.464958 (+ 17us) server_negotiation.cc:407] Connection header received
0809 01:56:43.465035 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:43.465037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:43.465090 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:43.465179 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:56:43.465413 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.465974 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.466772 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:43.466982 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:43.467057 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:43.467330 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:56:43.467741 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:56:43.467837 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:43.467902 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 01:56:43.467946 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 01:56:43.474155 787 raft_consensus.cc:1261] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 1 FOLLOWER]: Refusing update from remote peer a687d815ce1344cc85a2753c14c09d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:43.474192 595 raft_consensus.cc:1261] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [term 1 FOLLOWER]: Refusing update from remote peer a687d815ce1344cc85a2753c14c09d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:56:43.474980 1095 consensus_queue.cc:1035] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "debefd480d4748db8d270fd0c53b2d32" member_type: VOTER last_known_addr { host: "127.3.220.1" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:43.475165 1090 consensus_queue.cc:1035] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54899f0cfe744fafa3ed56d78caea360" member_type: VOTER last_known_addr { host: "127.3.220.2" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:56:43.483254 1103 mvcc.cc:205] Tried to move back new op lower bound from 7316441511821639680 to 7316441511698919424. Current Snapshot: MvccSnapshot[applied={T|T < 7316441511821639680}]
I20260809 01:56:43.490988 383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:43.491145 383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 01:56:43.493604 383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a34a401a4fc8418e93432a8818e816ea: Sending DeleteTablet for 3 replicas of tablet c6491c681ff646dc8cd096b481215114
I20260809 01:56:43.494465 569 tablet_service.cc:1514] Processing DeleteTablet for tablet c6491c681ff646dc8cd096b481215114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:43 UTC) from {username='slave'} at 127.0.0.1:51964
I20260809 01:56:43.494575 949 tablet_service.cc:1514] Processing DeleteTablet for tablet c6491c681ff646dc8cd096b481215114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:43 UTC) from {username='slave'} at 127.0.0.1:49458
I20260809 01:56:43.494601 760 tablet_service.cc:1514] Processing DeleteTablet for tablet c6491c681ff646dc8cd096b481215114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 01:56:43 UTC) from {username='slave'} at 127.0.0.1:46484
I20260809 01:56:43.495059 1119 tablet_replica.cc:330] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: stopping tablet replica
I20260809 01:56:43.495146 1118 tablet_replica.cc:330] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32: stopping tablet replica
I20260809 01:56:43.495431 1119 raft_consensus.cc:2229] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 1 LEADER]: Raft consensus shutting down.
I20260809 01:56:43.495430 1118 raft_consensus.cc:2229] T c6491c681ff646dc8cd096b481215114 P debefd480d4748db8d270fd0c53b2d32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:43.495620 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 478
I20260809 01:56:43.506732 1119 raft_consensus.cc:2258] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:43.507282 1121 tablet_replica.cc:330] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: stopping tablet replica
I20260809 01:56:43.507740 1121 raft_consensus.cc:2229] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 01:56:43.508183 1121 raft_consensus.cc:2258] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 01:56:43.509054 1119 ts_tablet_manager.cc:1905] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:43.509728 1121 ts_tablet_manager.cc:1905] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 01:56:43.512189 1119 ts_tablet_manager.cc:1918] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:43.512394 1119 log.cc:1196] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-2/wal/wals/c6491c681ff646dc8cd096b481215114
I20260809 01:56:43.512669 1121 ts_tablet_manager.cc:1918] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 01:56:43.512826 1119 ts_tablet_manager.cc:1939] T c6491c681ff646dc8cd096b481215114 P a687d815ce1344cc85a2753c14c09d5c: Deleting consensus metadata
I20260809 01:56:43.512862 1121 log.cc:1196] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Deleting WAL directory at /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786240338176169-3952-0/minicluster-data/ts-1/wal/wals/c6491c681ff646dc8cd096b481215114
I20260809 01:56:43.513262 1121 ts_tablet_manager.cc:1939] T c6491c681ff646dc8cd096b481215114 P 54899f0cfe744fafa3ed56d78caea360: Deleting consensus metadata
I20260809 01:56:43.514458 362 catalog_manager.cc:4841] TS a687d815ce1344cc85a2753c14c09d5c (127.3.220.3:35877): tablet c6491c681ff646dc8cd096b481215114 (table test-table [id=68ef8c0c08e54c9f8831e59e95d2c301]) successfully deleted
I20260809 01:56:43.514724 362 catalog_manager.cc:4841] TS 54899f0cfe744fafa3ed56d78caea360 (127.3.220.2:46873): tablet c6491c681ff646dc8cd096b481215114 (table test-table [id=68ef8c0c08e54c9f8831e59e95d2c301]) successfully deleted
W20260809 01:56:43.514955 364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.220.1:38339 (error 108)
I20260809 01:56:43.515883 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 668
I20260809 01:56:43.516064 1072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:43.515707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.1:38339 (local address 127.0.0.1:51968)
0809 01:56:43.515902 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 01:56:43.515975 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.1:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 01:56:43.520833 364 catalog_manager.cc:4584] TS debefd480d4748db8d270fd0c53b2d32 (127.3.220.1:38339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6491c681ff646dc8cd096b481215114: Network error: Client connection negotiation failed: client connection to 127.3.220.1:38339: connect: Connection refused (error 111)
I20260809 01:56:43.530560 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 861
I20260809 01:56:43.542055 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 338
2026-08-09T01:56:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1364 ms)
[----------] 6 tests from EncryptionPolicyTest (11626 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:43 dist-test-slave-gkw7 krb5kdc[1127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:43 dist-test-slave-gkw7 krb5kdc[1127](info): set up 2 sockets
Aug 09 01:56:43 dist-test-slave-gkw7 krb5kdc[1127](info): commencing operation
krb5kdc: starting...
W20260809 01:56:44.823666 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:44 dist-test-slave-gkw7 krb5kdc[1127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:44.953013 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:45.066473 1154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:45.066829 1154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:45.066936 1154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:45.071332 1154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:45.071413 1154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:45.071451 1154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:45.071499 1154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:45.071535 1154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:45.077404 1154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:45.078545 1154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:45.079627 1154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:45.088572 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:45.088546 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:45.088629 1154 server_base.cc:1034] running on GCE node
W20260809 01:56:45.088550 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:45.089457 1154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:45.090584 1154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:45.091753 1154 hybrid_clock.cc:648] HybridClock initialized: now 1786240605091735 us; error 31 us; skew 500 ppm
Aug 09 01:56:45 dist-test-slave-gkw7 krb5kdc[1127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:45.094611 1154 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:45.095541 1154 webserver.cc:460] Webserver started at http://127.0.0.1:44591/ using document root <none> and password file <none>
I20260809 01:56:45.096105 1154 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:45.096166 1154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:45.096402 1154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:45.098507 1154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "f1f6f434bcb748d4a7608bf594268e1e"
format_stamp: "Formatted at 2026-08-09 01:56:45 on dist-test-slave-gkw7"
server_key: "e78ec1f3d06fb789a3036185755e3e31"
server_key_iv: "8d7498ef0df9197beffdbee2d4ee3c6d"
server_key_version: "encryptionkey@0"
I20260809 01:56:45.099085 1154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "f1f6f434bcb748d4a7608bf594268e1e"
format_stamp: "Formatted at 2026-08-09 01:56:45 on dist-test-slave-gkw7"
server_key: "e78ec1f3d06fb789a3036185755e3e31"
server_key_iv: "8d7498ef0df9197beffdbee2d4ee3c6d"
server_key_version: "encryptionkey@0"
I20260809 01:56:45.102798 1154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:56:45.105146 1176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:45.106381 1154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:56:45.106526 1154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "f1f6f434bcb748d4a7608bf594268e1e"
format_stamp: "Formatted at 2026-08-09 01:56:45 on dist-test-slave-gkw7"
server_key: "e78ec1f3d06fb789a3036185755e3e31"
server_key_iv: "8d7498ef0df9197beffdbee2d4ee3c6d"
server_key_version: "encryptionkey@0"
I20260809 01:56:45.106621 1154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:45.147500 1154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:45.150591 1154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:45.150818 1154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:45.157797 1154 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38575
I20260809 01:56:45.157819 1260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38575 every 8 connection(s)
I20260809 01:56:45.158795 1154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:45.160164 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 1154
I20260809 01:56:45.160331 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:45.160627 3952 external_mini_cluster.cc:1428] Setting key cda4ebd9fa459da389294baf5f74141b
I20260809 01:56:45.161868 1266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:45.167563 1266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e: Bootstrap starting.
Aug 09 01:56:45 dist-test-slave-gkw7 krb5kdc[1127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 01:56:45.170013 1266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:45.170859 1266 log.cc:824] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:45.173041 1266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e: No bootstrap required, opened a new log
I20260809 01:56:45.175207 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:45.162147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36607 (local address 127.0.0.1:38575)
0809 01:56:45.162591 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:45.162606 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:45.162643 (+ 37us) server_negotiation.cc:407] Connection header received
0809 01:56:45.163443 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:45.163462 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:45.163807 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:45.164132 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:45.164970 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:45.166046 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:45.166917 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:45.167393 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:45.167565 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:45.170429 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:45.170435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:45.170447 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:45.170473 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:45.172224 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:45.172810 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:45.172814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:45.172818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:45.172928 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:45.173270 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:45.173273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:45.173274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:45.173752 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:45.173923 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:45.174253 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 01:56:45.174494 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":144,"threads_started":1}
I20260809 01:56:45.176815 1266 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f6f434bcb748d4a7608bf594268e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } }
I20260809 01:56:45.177052 1266 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:45.177114 1266 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f6f434bcb748d4a7608bf594268e1e, State: Initialized, Role: FOLLOWER
I20260809 01:56:45.177603 1266 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f6f434bcb748d4a7608bf594268e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } }
I20260809 01:56:45.177759 1266 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:45.177846 1266 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:45.177947 1266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:45.178192 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:45.178925 1266 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f6f434bcb748d4a7608bf594268e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } }
I20260809 01:56:45.179267 1266 leader_election.cc:302] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1f6f434bcb748d4a7608bf594268e1e; no voters:
I20260809 01:56:45.179656 1266 leader_election.cc:288] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:45.179752 1277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:45.179972 1277 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [term 1 LEADER]: Becoming Leader. State: Replica: f1f6f434bcb748d4a7608bf594268e1e, State: Running, Role: LEADER
I20260809 01:56:45.180800 1277 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f6f434bcb748d4a7608bf594268e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } }
I20260809 01:56:45.180994 1266 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:45.182781 1281 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1f6f434bcb748d4a7608bf594268e1e. Latest consensus state: current_term: 1 leader_uuid: "f1f6f434bcb748d4a7608bf594268e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f6f434bcb748d4a7608bf594268e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } } }
I20260809 01:56:45.182802 1280 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1f6f434bcb748d4a7608bf594268e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1f6f434bcb748d4a7608bf594268e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } } }
I20260809 01:56:45.182950 1281 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:45.182963 1280 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:45.183316 1303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:45.186908 1303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:45.189858 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:45.180387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49123 (local address 127.0.0.1:38575)
0809 01:56:45.180502 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:45.180507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:45.180709 (+ 202us) server_negotiation.cc:407] Connection header received
0809 01:56:45.180824 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:45.180827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:45.180875 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:45.180958 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:45.181942 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:45.182799 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:45.184015 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:45.184357 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:45.184470 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:45.186167 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:45.186171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:45.186174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:45.186200 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:45.187855 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:45.188428 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:45.188432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:45.188433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:45.188521 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:45.188948 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:45.188953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:45.188954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:45.189141 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:45.189230 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:45.189607 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 01:56:45.189699 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 01:56:45.193400 1303 catalog_manager.cc:1347] Generated new cluster ID: 804ad51ec8ba49d0b5191ac05ced4fb6
I20260809 01:56:45.193462 1303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:45.207250 1303 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:45.208489 1303 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:45.213729 1303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1f6f434bcb748d4a7608bf594268e1e: Generated new TSK 0
I20260809 01:56:45.214499 1303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:45.223709 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 1154
W20260809 01:56:45.232908 1124 connection.cc:535] client connection to 127.0.0.1:38575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38575): Connection reset by peer (error 104)
2026-08-09T01:56:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1676 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:45 dist-test-slave-gkw7 krb5kdc[1316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:45 dist-test-slave-gkw7 krb5kdc[1316](info): set up 2 sockets
Aug 09 01:56:45 dist-test-slave-gkw7 krb5kdc[1316](info): commencing operation
krb5kdc: starting...
W20260809 01:56:46.473690 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:46 dist-test-slave-gkw7 krb5kdc[1316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:46.604840 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:46.719655 1339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:46.719959 1339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:46.720033 1339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:46.724097 1339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:46.724186 1339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:46.724232 1339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:46.724287 1339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:46.724327 1339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:46.729614 1339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:46.730717 1339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:46.731758 1339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:46.740516 1347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:46.740497 1348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:46.740602 1339 server_base.cc:1034] running on GCE node
W20260809 01:56:46.740818 1352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:46.741397 1339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:46.742548 1339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:46.743745 1339 hybrid_clock.cc:648] HybridClock initialized: now 1786240606743722 us; error 32 us; skew 500 ppm
Aug 09 01:56:46 dist-test-slave-gkw7 krb5kdc[1316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:46.746934 1339 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:46.747879 1339 webserver.cc:460] Webserver started at http://127.0.0.1:34459/ using document root <none> and password file <none>
I20260809 01:56:46.748450 1339 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:46.748538 1339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:46.748775 1339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:46.750982 1339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "498336d8fbad4051972e752b3a78462c"
format_stamp: "Formatted at 2026-08-09 01:56:46 on dist-test-slave-gkw7"
server_key: "71868a87206abe98e90632bbfe66c453"
server_key_iv: "381496c6e375e9eba9961b599aabcdc3"
server_key_version: "encryptionkey@0"
I20260809 01:56:46.751610 1339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "498336d8fbad4051972e752b3a78462c"
format_stamp: "Formatted at 2026-08-09 01:56:46 on dist-test-slave-gkw7"
server_key: "71868a87206abe98e90632bbfe66c453"
server_key_iv: "381496c6e375e9eba9961b599aabcdc3"
server_key_version: "encryptionkey@0"
I20260809 01:56:46.755487 1339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 01:56:46.758080 1364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:46.759321 1339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:46.759472 1339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "498336d8fbad4051972e752b3a78462c"
format_stamp: "Formatted at 2026-08-09 01:56:46 on dist-test-slave-gkw7"
server_key: "71868a87206abe98e90632bbfe66c453"
server_key_iv: "381496c6e375e9eba9961b599aabcdc3"
server_key_version: "encryptionkey@0"
I20260809 01:56:46.759569 1339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:46.777190 1339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:46.780525 1339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:46.780735 1339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:46.787707 1339 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42549
I20260809 01:56:46.787748 1433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42549 every 8 connection(s)
I20260809 01:56:46.788673 1339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:46.791318 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 1339
I20260809 01:56:46.791456 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:46.791605 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:46.791790 3952 external_mini_cluster.cc:1428] Setting key 5baca0ad0a4094b2c32c1891d44cee79
I20260809 01:56:46.797760 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c: Bootstrap starting.
I20260809 01:56:46.800001 1437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:46.800462 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:46.793138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42001 (local address 127.0.0.1:42549)
0809 01:56:46.793632 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:46.793648 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:46.793690 (+ 42us) server_negotiation.cc:407] Connection header received
0809 01:56:46.794308 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:46.794324 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:46.794642 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:46.794972 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:46.795373 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:46.796348 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:46.797252 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:46.797782 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:46.797962 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:46.798850 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:46.798855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:46.798868 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:46.799237 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:46.799289 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:46.799311 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:46.799448 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:46.799604 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 01:56:46.799744 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
I20260809 01:56:46.800877 1437 log.cc:824] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:46.802985 1437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c: No bootstrap required, opened a new log
I20260809 01:56:46.803433 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:46.806823 1437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498336d8fbad4051972e752b3a78462c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42549 } }
I20260809 01:56:46.807065 1437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:46.807156 1437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498336d8fbad4051972e752b3a78462c, State: Initialized, Role: FOLLOWER
I20260809 01:56:46.807575 1437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498336d8fbad4051972e752b3a78462c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42549 } }
I20260809 01:56:46.807732 1437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:46.807813 1437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:46.807917 1437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:46.808928 1437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498336d8fbad4051972e752b3a78462c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42549 } }
I20260809 01:56:46.809285 1437 leader_election.cc:302] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 498336d8fbad4051972e752b3a78462c; no voters:
I20260809 01:56:46.809641 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:46.805434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45631 (local address 127.0.0.1:42549)
0809 01:56:46.805586 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:46.805590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:46.805747 (+ 157us) server_negotiation.cc:407] Connection header received
0809 01:56:46.805869 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:46.805875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:46.805921 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:46.806004 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:46.806277 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:46.806768 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:46.807828 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:46.808164 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:46.808276 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:46.809120 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:46.809124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:46.809127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:46.809139 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:46.809170 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:46.809180 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:46.809274 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:46.809443 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 01:56:46.809478 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 01:56:46.809644 1437 leader_election.cc:288] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:46.809718 1450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:46.810086 1450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [term 1 LEADER]: Becoming Leader. State: Replica: 498336d8fbad4051972e752b3a78462c, State: Running, Role: LEADER
I20260809 01:56:46.810441 1450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498336d8fbad4051972e752b3a78462c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42549 } }
I20260809 01:56:46.811123 1437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:46.811695 1451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "498336d8fbad4051972e752b3a78462c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498336d8fbad4051972e752b3a78462c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42549 } } }
I20260809 01:56:46.811828 1451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:46.812201 1452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 498336d8fbad4051972e752b3a78462c. Latest consensus state: current_term: 1 leader_uuid: "498336d8fbad4051972e752b3a78462c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498336d8fbad4051972e752b3a78462c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42549 } } }
I20260809 01:56:46.812299 1452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:46.813599 1459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:46.817024 1459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:46.822590 1459 catalog_manager.cc:1347] Generated new cluster ID: 1a7113a703e44782b398dec58174c4d3
I20260809 01:56:46.822661 1459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:46.836707 1459 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:46.838008 1459 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:46.843144 1459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 498336d8fbad4051972e752b3a78462c: Generated new TSK 0
I20260809 01:56:46.843815 1459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:46.850425 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 1339
W20260809 01:56:46.865046 1312 connection.cc:535] client connection to 127.0.0.1:42549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42549): Connection reset by peer (error 104)
2026-08-09T01:56:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1632 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:46 dist-test-slave-gkw7 krb5kdc[1482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:46 dist-test-slave-gkw7 krb5kdc[1482](info): set up 2 sockets
Aug 09 01:56:46 dist-test-slave-gkw7 krb5kdc[1482](info): commencing operation
krb5kdc: starting...
W20260809 01:56:48.079303 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:48 dist-test-slave-gkw7 krb5kdc[1482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:48.210927 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:48.322270 1499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:48.322595 1499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:48.322697 1499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:48.326689 1499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:48.326762 1499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:48.326800 1499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:48.326853 1499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:48.326893 1499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:48.332054 1499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:48.333068 1499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:48.334120 1499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:48.342427 1507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:48.342397 1509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:48.342471 1506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:48.342764 1499 server_base.cc:1034] running on GCE node
I20260809 01:56:48.343307 1499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:48.344388 1499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:48.345613 1499 hybrid_clock.cc:648] HybridClock initialized: now 1786240608345558 us; error 55 us; skew 500 ppm
Aug 09 01:56:48 dist-test-slave-gkw7 krb5kdc[1482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:48.352929 1499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:48.353873 1499 webserver.cc:460] Webserver started at http://127.0.0.1:44053/ using document root <none> and password file <none>
I20260809 01:56:48.354418 1499 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:48.354480 1499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:48.354655 1499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:48.356662 1499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "0b34b205ed5449319dda864e8f4e2740"
format_stamp: "Formatted at 2026-08-09 01:56:48 on dist-test-slave-gkw7"
server_key: "16728a2d265dcad9e547a298c52127ae"
server_key_iv: "527d3a85870ea604cf51029488438ecc"
server_key_version: "encryptionkey@0"
I20260809 01:56:48.357247 1499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "0b34b205ed5449319dda864e8f4e2740"
format_stamp: "Formatted at 2026-08-09 01:56:48 on dist-test-slave-gkw7"
server_key: "16728a2d265dcad9e547a298c52127ae"
server_key_iv: "527d3a85870ea604cf51029488438ecc"
server_key_version: "encryptionkey@0"
I20260809 01:56:48.360951 1499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 01:56:48.363449 1522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:48.364675 1499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 01:56:48.364812 1499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "0b34b205ed5449319dda864e8f4e2740"
format_stamp: "Formatted at 2026-08-09 01:56:48 on dist-test-slave-gkw7"
server_key: "16728a2d265dcad9e547a298c52127ae"
server_key_iv: "527d3a85870ea604cf51029488438ecc"
server_key_version: "encryptionkey@0"
I20260809 01:56:48.364920 1499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:48.369621 1499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:48.372661 1499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:48.372865 1499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:48.379945 1499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40909
I20260809 01:56:48.379966 1588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40909 every 8 connection(s)
I20260809 01:56:48.380848 1499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:48.383639 1591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:48.387450 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 1499
I20260809 01:56:48.387615 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:48.387890 3952 external_mini_cluster.cc:1428] Setting key 3c58a0070c77e0f3cf6d88b2ef0b0d84
I20260809 01:56:48.389374 1591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740: Bootstrap starting.
I20260809 01:56:48.392062 1591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:48.392956 1591 log.cc:824] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:48.393415 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:48.389256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50399 (local address 127.0.0.1:40909)
0809 01:56:48.389735 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:48.389753 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:48.389795 (+ 42us) server_negotiation.cc:407] Connection header received
0809 01:56:48.390603 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:48.390626 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:48.390994 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:48.391343 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:48.392028 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:48.392036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:48.392052 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:48.392064 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:48.392129 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:48.392159 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:48.392277 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:48.392364 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 01:56:48.392512 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260809 01:56:48.395093 1591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740: No bootstrap required, opened a new log
I20260809 01:56:48.396317 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:48.399900 1591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b34b205ed5449319dda864e8f4e2740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } }
I20260809 01:56:48.400159 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:48.398639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33723 (local address 127.0.0.1:40909)
0809 01:56:48.398748 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:48.398752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:48.398926 (+ 174us) server_negotiation.cc:407] Connection header received
0809 01:56:48.399043 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:48.399045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:48.399071 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:48.399133 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:48.399779 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:48.399781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:48.399783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:48.399793 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:48.399816 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:48.399827 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:48.399894 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:48.400010 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 01:56:48.400038 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:56:48.400192 1591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:48.400326 1591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b34b205ed5449319dda864e8f4e2740, State: Initialized, Role: FOLLOWER
I20260809 01:56:48.400775 1591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b34b205ed5449319dda864e8f4e2740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } }
I20260809 01:56:48.400935 1591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:48.401010 1591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:48.401115 1591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:48.402091 1591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b34b205ed5449319dda864e8f4e2740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } }
I20260809 01:56:48.402429 1591 leader_election.cc:302] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b34b205ed5449319dda864e8f4e2740; no voters:
I20260809 01:56:48.402741 1591 leader_election.cc:288] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:48.402802 1606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:48.403047 1606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [term 1 LEADER]: Becoming Leader. State: Replica: 0b34b205ed5449319dda864e8f4e2740, State: Running, Role: LEADER
I20260809 01:56:48.403383 1606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b34b205ed5449319dda864e8f4e2740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } }
I20260809 01:56:48.403916 1591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:48.405097 1609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b34b205ed5449319dda864e8f4e2740. Latest consensus state: current_term: 1 leader_uuid: "0b34b205ed5449319dda864e8f4e2740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b34b205ed5449319dda864e8f4e2740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } } }
I20260809 01:56:48.405239 1609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:48.405608 1608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b34b205ed5449319dda864e8f4e2740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b34b205ed5449319dda864e8f4e2740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } } }
I20260809 01:56:48.405696 1608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:48.405702 1621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:48.408545 1621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:48.414075 1621 catalog_manager.cc:1347] Generated new cluster ID: 3140b2ceb58e4c6d9743d7a0bc1fefd7
I20260809 01:56:48.414137 1621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:48.438858 1621 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:48.439203 1621 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:48.446132 1621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b34b205ed5449319dda864e8f4e2740: Generated new TSK 0
I20260809 01:56:48.446851 1621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:48.461288 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 1499
2026-08-09T01:56:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:48 dist-test-slave-gkw7 krb5kdc[1637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:48 dist-test-slave-gkw7 krb5kdc[1637](info): set up 2 sockets
Aug 09 01:56:48 dist-test-slave-gkw7 krb5kdc[1637](info): commencing operation
krb5kdc: starting...
W20260809 01:56:49.717835 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:49 dist-test-slave-gkw7 krb5kdc[1637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:49.844323 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:49.957047 1659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:49.957381 1659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:49.957480 1659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:49.961696 1659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:49.961781 1659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:49.961822 1659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:49.961889 1659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:49.961928 1659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:49.967046 1659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:49.968091 1659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:49.969095 1659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:49.978302 1671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:49.978456 1659 server_base.cc:1034] running on GCE node
W20260809 01:56:49.978245 1667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:49.978263 1668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:49.979100 1659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:49.980204 1659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:49.981369 1659 hybrid_clock.cc:648] HybridClock initialized: now 1786240609981349 us; error 29 us; skew 500 ppm
Aug 09 01:56:49 dist-test-slave-gkw7 krb5kdc[1637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:49.984118 1659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:49.984970 1659 webserver.cc:460] Webserver started at http://127.0.0.1:38675/ using document root <none> and password file <none>
I20260809 01:56:49.985586 1659 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:49.985673 1659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:49.985904 1659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:49.987916 1659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "4fa60fc916e748cdaf26320c659cf0a7"
format_stamp: "Formatted at 2026-08-09 01:56:49 on dist-test-slave-gkw7"
server_key: "331c6824356472afe03cd492fae9600b"
server_key_iv: "3ceb24abd9baa967481a32906354b85c"
server_key_version: "encryptionkey@0"
I20260809 01:56:49.988469 1659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "4fa60fc916e748cdaf26320c659cf0a7"
format_stamp: "Formatted at 2026-08-09 01:56:49 on dist-test-slave-gkw7"
server_key: "331c6824356472afe03cd492fae9600b"
server_key_iv: "3ceb24abd9baa967481a32906354b85c"
server_key_version: "encryptionkey@0"
I20260809 01:56:49.992190 1659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 01:56:49.994585 1678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:49.995785 1659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:49.995913 1659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "4fa60fc916e748cdaf26320c659cf0a7"
format_stamp: "Formatted at 2026-08-09 01:56:49 on dist-test-slave-gkw7"
server_key: "331c6824356472afe03cd492fae9600b"
server_key_iv: "3ceb24abd9baa967481a32906354b85c"
server_key_version: "encryptionkey@0"
I20260809 01:56:49.996013 1659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:50.015373 1659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:50.018180 1659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:50.018378 1659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:50.025305 1659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37261
I20260809 01:56:50.025398 1751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37261 every 8 connection(s)
I20260809 01:56:50.026283 1659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:50.029002 1753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:50.030862 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 1659
I20260809 01:56:50.031024 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:50.031311 3952 external_mini_cluster.cc:1428] Setting key 1936420e1f4e5885ca16feb8d0c34a21
I20260809 01:56:50.035701 1753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7: Bootstrap starting.
I20260809 01:56:50.038255 1753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:50.039193 1753 log.cc:824] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:56:50 dist-test-slave-gkw7 krb5kdc[1637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 01:56:50.041520 1753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7: No bootstrap required, opened a new log
I20260809 01:56:50.046059 1753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa60fc916e748cdaf26320c659cf0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260809 01:56:50.046265 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:50.032679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52409 (local address 127.0.0.1:37261)
0809 01:56:50.033083 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:50.033101 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:50.033139 (+ 38us) server_negotiation.cc:407] Connection header received
0809 01:56:50.033815 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:50.033833 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:50.034166 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:50.034444 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:50.035392 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:50.036453 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:50.037668 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:50.038116 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:50.038237 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:50.041014 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:50.041021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:50.041036 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:50.041066 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:50.043327 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:50.043924 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:50.043928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:50.043932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:50.044043 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:50.044341 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:50.044344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:50.044345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:50.044707 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:50.044881 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:50.045158 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 01:56:50.045462 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":135,"threads_started":1}
I20260809 01:56:50.046301 1753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:50.046367 1753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa60fc916e748cdaf26320c659cf0a7, State: Initialized, Role: FOLLOWER
I20260809 01:56:50.046844 1753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa60fc916e748cdaf26320c659cf0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260809 01:56:50.046985 1753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:50.047035 1753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:50.047116 1753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:50.048041 1753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa60fc916e748cdaf26320c659cf0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260809 01:56:50.048399 1753 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fa60fc916e748cdaf26320c659cf0a7; no voters:
I20260809 01:56:50.048751 1753 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:50.048857 1759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:50.049072 1759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [term 1 LEADER]: Becoming Leader. State: Replica: 4fa60fc916e748cdaf26320c659cf0a7, State: Running, Role: LEADER
I20260809 01:56:50.049167 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:50.049422 1759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa60fc916e748cdaf26320c659cf0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260809 01:56:50.050086 1753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:50.051002 1761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa60fc916e748cdaf26320c659cf0a7. Latest consensus state: current_term: 1 leader_uuid: "4fa60fc916e748cdaf26320c659cf0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa60fc916e748cdaf26320c659cf0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } } }
I20260809 01:56:50.051151 1761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:50.051507 1760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fa60fc916e748cdaf26320c659cf0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa60fc916e748cdaf26320c659cf0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } } }
I20260809 01:56:50.051587 1760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:50.051571 1774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:50.055119 1774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:50.061141 1774 catalog_manager.cc:1347] Generated new cluster ID: 3867ca19e8dd486da8ff4b0aadbb8846
I20260809 01:56:50.061210 1774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:50.061884 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:50.052446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58359 (local address 127.0.0.1:37261)
0809 01:56:50.052549 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:50.052553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:50.053048 (+ 495us) server_negotiation.cc:407] Connection header received
0809 01:56:50.053217 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:50.053220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:50.053266 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:50.053351 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:50.054202 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:50.054912 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:50.056042 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:50.056400 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:50.056514 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:50.058181 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:50.058185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:50.058188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:50.058213 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:50.059922 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:50.060336 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:50.060340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:50.060342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:50.060433 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:50.060712 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:50.060715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:50.060717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:50.060933 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:50.061061 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:50.061646 (+ 585us) server_negotiation.cc:299] Negotiation successful
0809 01:56:50.061760 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 01:56:50.072125 1774 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:50.073026 1774 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:50.079435 1774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fa60fc916e748cdaf26320c659cf0a7: Generated new TSK 0
I20260809 01:56:50.080091 1774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:50.087038 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 1659
2026-08-09T01:56:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1626 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:50 dist-test-slave-gkw7 krb5kdc[1794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:50 dist-test-slave-gkw7 krb5kdc[1794](info): set up 2 sockets
Aug 09 01:56:50 dist-test-slave-gkw7 krb5kdc[1794](info): commencing operation
krb5kdc: starting...
W20260809 01:56:51.305433 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:51 dist-test-slave-gkw7 krb5kdc[1794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:51.436726 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:51.549113 1816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:51.549378 1816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:51.549420 1816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:51.553604 1816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:51.553689 1816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:51.553735 1816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:51.553800 1816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:51.553856 1816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:51.559369 1816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:51.560448 1816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:51.561498 1816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:51.570091 1823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:51.570125 1828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:51.570134 1824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:51.570185 1816 server_base.cc:1034] running on GCE node
I20260809 01:56:51.570928 1816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:51.571947 1816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:51.573134 1816 hybrid_clock.cc:648] HybridClock initialized: now 1786240611573107 us; error 37 us; skew 500 ppm
Aug 09 01:56:51 dist-test-slave-gkw7 krb5kdc[1794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:51.576015 1816 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:51.576973 1816 webserver.cc:460] Webserver started at http://127.0.0.1:43339/ using document root <none> and password file <none>
I20260809 01:56:51.577597 1816 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:51.577690 1816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:51.577927 1816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:51.580114 1816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "84239fad542f458db94cd8064c40b093"
format_stamp: "Formatted at 2026-08-09 01:56:51 on dist-test-slave-gkw7"
server_key: "f506dda343e8e1e4a1915836e99c5090"
server_key_iv: "556e1863a2f93743e877b37956426640"
server_key_version: "encryptionkey@0"
I20260809 01:56:51.580695 1816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "84239fad542f458db94cd8064c40b093"
format_stamp: "Formatted at 2026-08-09 01:56:51 on dist-test-slave-gkw7"
server_key: "f506dda343e8e1e4a1915836e99c5090"
server_key_iv: "556e1863a2f93743e877b37956426640"
server_key_version: "encryptionkey@0"
I20260809 01:56:51.584507 1816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 01:56:51.587020 1837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:51.588277 1816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 01:56:51.588441 1816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "84239fad542f458db94cd8064c40b093"
format_stamp: "Formatted at 2026-08-09 01:56:51 on dist-test-slave-gkw7"
server_key: "f506dda343e8e1e4a1915836e99c5090"
server_key_iv: "556e1863a2f93743e877b37956426640"
server_key_version: "encryptionkey@0"
I20260809 01:56:51.588562 1816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:51.602437 1816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:51.605276 1816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:51.605494 1816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:51.613005 1816 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40855
I20260809 01:56:51.613014 1913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40855 every 8 connection(s)
I20260809 01:56:51.614141 1816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:51.617023 1914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:51.622715 1914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093: Bootstrap starting.
I20260809 01:56:51.623502 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 1816
I20260809 01:56:51.623679 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:51.623955 3952 external_mini_cluster.cc:1428] Setting key df2cf78969c2cbce8bbb721cc3b67aba
I20260809 01:56:51.625521 1914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:51.626304 1914 log.cc:824] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:51.628193 1914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093: No bootstrap required, opened a new log
I20260809 01:56:51.631807 1914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84239fad542f458db94cd8064c40b093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260809 01:56:51.632061 1914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:51.632179 1914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84239fad542f458db94cd8064c40b093, State: Initialized, Role: FOLLOWER
I20260809 01:56:51.632644 1914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84239fad542f458db94cd8064c40b093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260809 01:56:51.632771 1914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:51.632870 1914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:51.632998 1914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:51.633247 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:51.625166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50471 (local address 127.0.0.1:40855)
0809 01:56:51.625645 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:51.625684 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:51.625736 (+ 52us) server_negotiation.cc:407] Connection header received
0809 01:56:51.626395 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:51.626413 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:51.626809 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:51.627149 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:51.627594 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:51.628646 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:51.629618 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:51.630099 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:51.630292 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:51.631252 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:51.631258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:51.631271 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:51.631684 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:51.631743 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:51.631769 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:51.631888 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:51.632047 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 01:56:51.632285 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20260809 01:56:51.634030 1914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84239fad542f458db94cd8064c40b093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260809 01:56:51.634377 1914 leader_election.cc:302] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84239fad542f458db94cd8064c40b093; no voters:
I20260809 01:56:51.634713 1914 leader_election.cc:288] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:51.634785 1919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:51.635020 1919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [term 1 LEADER]: Becoming Leader. State: Replica: 84239fad542f458db94cd8064c40b093, State: Running, Role: LEADER
I20260809 01:56:51.635807 1919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84239fad542f458db94cd8064c40b093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } }
I20260809 01:56:51.636076 1914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:51.637015 1919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84239fad542f458db94cd8064c40b093. Latest consensus state: current_term: 1 leader_uuid: "84239fad542f458db94cd8064c40b093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84239fad542f458db94cd8064c40b093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } } }
I20260809 01:56:51.637141 1919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:51.637497 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:51.637523 1931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:51.639065 1920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84239fad542f458db94cd8064c40b093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84239fad542f458db94cd8064c40b093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40855 } } }
I20260809 01:56:51.639819 1920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:51.641076 1931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:51.644881 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:51.639946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47611 (local address 127.0.0.1:40855)
0809 01:56:51.640052 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:51.640057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:51.640310 (+ 253us) server_negotiation.cc:407] Connection header received
0809 01:56:51.640518 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:51.640521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:51.640586 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:51.640686 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:51.640999 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:51.641764 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:51.642927 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:51.643307 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:51.643432 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:51.644356 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:51.644360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:51.644363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:51.644378 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:51.644409 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:51.644424 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:51.644511 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:51.644636 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 01:56:51.644704 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 01:56:51.648129 1931 catalog_manager.cc:1347] Generated new cluster ID: a5efac3008c44dffa390566b038df2a5
I20260809 01:56:51.648213 1931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:51.659134 1931 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:51.660157 1931 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:51.677618 1931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84239fad542f458db94cd8064c40b093: Generated new TSK 0
I20260809 01:56:51.678488 1931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:51.682828 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 1816
2026-08-09T01:56:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1592 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:51 dist-test-slave-gkw7 krb5kdc[1960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:51 dist-test-slave-gkw7 krb5kdc[1960](info): set up 2 sockets
Aug 09 01:56:51 dist-test-slave-gkw7 krb5kdc[1960](info): commencing operation
krb5kdc: starting...
W20260809 01:56:52.921203 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:53 dist-test-slave-gkw7 krb5kdc[1960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:53.052346 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:53.165063 1980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:53.165337 1980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:53.165412 1980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:53.169283 1980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:53.169363 1980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:53.169409 1980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:53.169458 1980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:53.169493 1980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:53.174618 1980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:53.175660 1980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:53.176648 1980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:56:53.185474 1980 server_base.cc:1034] running on GCE node
W20260809 01:56:53.185489 1989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:53.185489 1992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:53.185689 1995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:53.186302 1980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:53.187341 1980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:53.188503 1980 hybrid_clock.cc:648] HybridClock initialized: now 1786240613188474 us; error 41 us; skew 500 ppm
Aug 09 01:56:53 dist-test-slave-gkw7 krb5kdc[1960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:53.191263 1980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:53.192090 1980 webserver.cc:460] Webserver started at http://127.0.0.1:37073/ using document root <none> and password file <none>
I20260809 01:56:53.192602 1980 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:53.192659 1980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:53.192823 1980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:53.194875 1980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "b9b8560e9bb14931acdc27e9891cf01b"
format_stamp: "Formatted at 2026-08-09 01:56:53 on dist-test-slave-gkw7"
server_key: "74a8554c422c27814696fc256f49221c"
server_key_iv: "f6bff5ed67024b7d72c75bea1e771580"
server_key_version: "encryptionkey@0"
I20260809 01:56:53.195405 1980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "b9b8560e9bb14931acdc27e9891cf01b"
format_stamp: "Formatted at 2026-08-09 01:56:53 on dist-test-slave-gkw7"
server_key: "74a8554c422c27814696fc256f49221c"
server_key_iv: "f6bff5ed67024b7d72c75bea1e771580"
server_key_version: "encryptionkey@0"
I20260809 01:56:53.199049 1980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:56:53.201349 2005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:53.202581 1980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:53.202720 1980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "b9b8560e9bb14931acdc27e9891cf01b"
format_stamp: "Formatted at 2026-08-09 01:56:53 on dist-test-slave-gkw7"
server_key: "74a8554c422c27814696fc256f49221c"
server_key_iv: "f6bff5ed67024b7d72c75bea1e771580"
server_key_version: "encryptionkey@0"
I20260809 01:56:53.202831 1980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:53.207496 1980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:53.210721 1980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:53.210920 1980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:53.217698 1980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40151
I20260809 01:56:53.217748 2083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40151 every 8 connection(s)
I20260809 01:56:53.218643 1980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:53.219229 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 1980
I20260809 01:56:53.219398 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:53.219695 3952 external_mini_cluster.cc:1428] Setting key 5e827f6668060dab6cbcd60f45630836
I20260809 01:56:53.221817 2084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:53.224803 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:53.221037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60255 (local address 127.0.0.1:40151)
0809 01:56:53.221486 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:53.221501 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:53.221577 (+ 76us) server_negotiation.cc:407] Connection header received
0809 01:56:53.222280 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:53.222298 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:53.222579 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:53.222913 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:53.223642 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:53.223647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:53.223666 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:53.223676 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:53.223727 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:53.223746 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:53.223894 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:53.223979 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 01:56:53.224121 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":153,"threads_started":1}
I20260809 01:56:53.227705 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:53.228205 2084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b: Bootstrap starting.
I20260809 01:56:53.230695 2084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:53.231375 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:53.229774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42027 (local address 127.0.0.1:40151)
0809 01:56:53.229907 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:53.229911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:53.230122 (+ 211us) server_negotiation.cc:407] Connection header received
0809 01:56:53.230243 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:53.230245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:53.230272 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:53.230340 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:53.231018 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:53.231020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:53.231022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:53.231032 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:53.231054 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:53.231064 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:53.231129 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:53.231230 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 01:56:53.231258 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 01:56:53.231633 2084 log.cc:824] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:53.233686 2084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b: No bootstrap required, opened a new log
I20260809 01:56:53.237219 2084 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b8560e9bb14931acdc27e9891cf01b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260809 01:56:53.237425 2084 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:53.237459 2084 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b8560e9bb14931acdc27e9891cf01b, State: Initialized, Role: FOLLOWER
I20260809 01:56:53.237882 2084 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b8560e9bb14931acdc27e9891cf01b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260809 01:56:53.238000 2084 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:53.238044 2084 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:53.238111 2084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:53.238883 2084 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b8560e9bb14931acdc27e9891cf01b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260809 01:56:53.239162 2084 leader_election.cc:302] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9b8560e9bb14931acdc27e9891cf01b; no voters:
I20260809 01:56:53.239398 2084 leader_election.cc:288] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:53.239511 2099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:53.239741 2099 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [term 1 LEADER]: Becoming Leader. State: Replica: b9b8560e9bb14931acdc27e9891cf01b, State: Running, Role: LEADER
I20260809 01:56:53.240098 2099 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b8560e9bb14931acdc27e9891cf01b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260809 01:56:53.240631 2084 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:53.242023 2100 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9b8560e9bb14931acdc27e9891cf01b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b8560e9bb14931acdc27e9891cf01b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } } }
I20260809 01:56:53.242152 2100 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:53.242581 2102 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9b8560e9bb14931acdc27e9891cf01b. Latest consensus state: current_term: 1 leader_uuid: "b9b8560e9bb14931acdc27e9891cf01b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b8560e9bb14931acdc27e9891cf01b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } } }
I20260809 01:56:53.242687 2102 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b [sys.catalog]: This master's current role is: LEADER
W20260809 01:56:53.245428 2122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:56:53.245517 2122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:56:53.245635 2117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:53.246522 2117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:53.252393 2117 catalog_manager.cc:1347] Generated new cluster ID: 9de0b179363f4d08bb520b3a5967119a
I20260809 01:56:53.252473 2117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:53.268314 2117 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:53.268668 2117 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:53.280305 2117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9b8560e9bb14931acdc27e9891cf01b: Generated new TSK 0
I20260809 01:56:53.281203 2117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:53.288658 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 1980
2026-08-09T01:56:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1608 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:53 dist-test-slave-gkw7 krb5kdc[2130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:53 dist-test-slave-gkw7 krb5kdc[2130](info): set up 2 sockets
Aug 09 01:56:53 dist-test-slave-gkw7 krb5kdc[2130](info): commencing operation
krb5kdc: starting...
W20260809 01:56:54.528009 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:54 dist-test-slave-gkw7 krb5kdc[2130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:54.662825 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:54.782055 2152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:54.782382 2152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:54.782481 2152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:54.786520 2152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:54.786608 2152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:54.786655 2152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:54.786691 2152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:54.786746 2152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:54.792189 2152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:54.793282 2152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:54.794411 2152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:54.803174 2162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:54.803244 2152 server_base.cc:1034] running on GCE node
W20260809 01:56:54.803226 2164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:54.803192 2161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:54.803962 2152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:54.805004 2152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:54.806164 2152 hybrid_clock.cc:648] HybridClock initialized: now 1786240614806134 us; error 40 us; skew 500 ppm
Aug 09 01:56:54 dist-test-slave-gkw7 krb5kdc[2130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:54.808897 2152 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:54.809851 2152 webserver.cc:460] Webserver started at http://127.0.0.1:36067/ using document root <none> and password file <none>
I20260809 01:56:54.810374 2152 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:54.810433 2152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:54.810602 2152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:54.812611 2152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "302ae046a492483d863b32b124f133ba"
format_stamp: "Formatted at 2026-08-09 01:56:54 on dist-test-slave-gkw7"
server_key: "a0b9f899c4e6f5150414d1b67afd1c42"
server_key_iv: "62fd3ed786e8d32414b857d68aa8b608"
server_key_version: "encryptionkey@0"
I20260809 01:56:54.813138 2152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "302ae046a492483d863b32b124f133ba"
format_stamp: "Formatted at 2026-08-09 01:56:54 on dist-test-slave-gkw7"
server_key: "a0b9f899c4e6f5150414d1b67afd1c42"
server_key_iv: "62fd3ed786e8d32414b857d68aa8b608"
server_key_version: "encryptionkey@0"
I20260809 01:56:54.816895 2152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:56:54.819514 2173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:54.820988 2152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:56:54.821121 2152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "302ae046a492483d863b32b124f133ba"
format_stamp: "Formatted at 2026-08-09 01:56:54 on dist-test-slave-gkw7"
server_key: "a0b9f899c4e6f5150414d1b67afd1c42"
server_key_iv: "62fd3ed786e8d32414b857d68aa8b608"
server_key_version: "encryptionkey@0"
I20260809 01:56:54.821270 2152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:54.846355 2152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:54.849414 2152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:54.849665 2152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:54.856817 2152 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34895
I20260809 01:56:54.856877 2252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34895 every 8 connection(s)
I20260809 01:56:54.857903 2152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:54.860345 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 2152
I20260809 01:56:54.860496 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:54.860802 3952 external_mini_cluster.cc:1428] Setting key 8a93d2b3eeccdf3f2e3efb9c50d73668
I20260809 01:56:54.861001 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:54.867498 2253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba: Bootstrap starting.
I20260809 01:56:54.869843 2253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba: Neither blocks nor log segments found. Creating new log.
Aug 09 01:56:54 dist-test-slave-gkw7 krb5kdc[2130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 01:56:54.870638 2253 log.cc:824] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:54.872525 2253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba: No bootstrap required, opened a new log
I20260809 01:56:54.875430 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:54.862360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:50387 (local address 127.0.0.1:34895)
0809 01:56:54.862838 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:54.862854 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:54.862890 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:56:54.863816 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:54.863834 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:54.864189 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:54.864531 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:54.865347 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:54.866446 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:54.867335 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:54.867863 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:54.868046 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:54.870699 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:54.870704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:54.870718 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:54.870745 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:54.872525 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:54.873085 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:54.873090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:54.873095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:54.873212 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:54.873584 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:54.873589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:54.873596 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:54.873969 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:54.874204 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:54.874504 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 01:56:54.874723 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":172,"threads_started":1}
I20260809 01:56:54.876286 2253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302ae046a492483d863b32b124f133ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } }
I20260809 01:56:54.876600 2253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:54.876673 2253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302ae046a492483d863b32b124f133ba, State: Initialized, Role: FOLLOWER
I20260809 01:56:54.877130 2253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302ae046a492483d863b32b124f133ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } }
I20260809 01:56:54.877315 2253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:54.877404 2253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:54.877513 2253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:54.878610 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:54.878731 2253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302ae046a492483d863b32b124f133ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } }
I20260809 01:56:54.879091 2253 leader_election.cc:302] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 302ae046a492483d863b32b124f133ba; no voters:
I20260809 01:56:54.879449 2253 leader_election.cc:288] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:54.879522 2260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:54.879755 2260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [term 1 LEADER]: Becoming Leader. State: Replica: 302ae046a492483d863b32b124f133ba, State: Running, Role: LEADER
I20260809 01:56:54.880118 2260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302ae046a492483d863b32b124f133ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } }
I20260809 01:56:54.880764 2253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:54.881386 2260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 302ae046a492483d863b32b124f133ba. Latest consensus state: current_term: 1 leader_uuid: "302ae046a492483d863b32b124f133ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302ae046a492483d863b32b124f133ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } } }
I20260809 01:56:54.881614 2260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:54.881862 2261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "302ae046a492483d863b32b124f133ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302ae046a492483d863b32b124f133ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34895 } } }
I20260809 01:56:54.882002 2261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:54.882095 2281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:54.885624 2281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:54.891031 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:54.880818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:47549 (local address 127.0.0.1:34895)
0809 01:56:54.880958 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:54.880962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:54.881388 (+ 426us) server_negotiation.cc:407] Connection header received
0809 01:56:54.881619 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:54.881624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:54.881698 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:54.881759 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:54.882741 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:54.883550 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:54.884736 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:54.885101 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:54.885207 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:54.887088 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:54.887092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:54.887094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:54.887120 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:54.888892 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:54.889484 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:54.889487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:54.889489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:54.889610 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:54.889990 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:54.889994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:54.889995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:54.890180 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:54.890345 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:54.890699 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 01:56:54.890822 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 01:56:54.892309 2281 catalog_manager.cc:1347] Generated new cluster ID: d80225b5c56f4a898dbbcb94d854d919
I20260809 01:56:54.892364 2281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:54.899399 2281 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:54.900576 2281 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:54.904310 2281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 302ae046a492483d863b32b124f133ba: Generated new TSK 0
I20260809 01:56:54.904989 2281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:54.909893 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 2152
W20260809 01:56:54.921433 2128 connection.cc:535] client connection to 127.0.0.1:34895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34895): Connection reset by peer (error 104)
2026-08-09T01:56:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1619 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:54 dist-test-slave-gkw7 krb5kdc[2297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:54 dist-test-slave-gkw7 krb5kdc[2297](info): set up 2 sockets
Aug 09 01:56:54 dist-test-slave-gkw7 krb5kdc[2297](info): commencing operation
krb5kdc: starting...
W20260809 01:56:56.146612 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:56 dist-test-slave-gkw7 krb5kdc[2297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:56.277488 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:56.390479 2315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:56.390837 2315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:56.390962 2315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:56.394937 2315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:56.394987 2315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:56.395006 2315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:56.395040 2315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:56.395056 2315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:56.400457 2315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:56.401590 2315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:56.402652 2315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:56.411319 2324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:56.411319 2325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:56.411337 2327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:56.411691 2315 server_base.cc:1034] running on GCE node
I20260809 01:56:56.412372 2315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:56.413695 2315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:56.414872 2315 hybrid_clock.cc:648] HybridClock initialized: now 1786240616414840 us; error 47 us; skew 500 ppm
Aug 09 01:56:56 dist-test-slave-gkw7 krb5kdc[2297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:56.417848 2315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:56.418798 2315 webserver.cc:460] Webserver started at http://127.0.0.1:40395/ using document root <none> and password file <none>
I20260809 01:56:56.419430 2315 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:56.419492 2315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:56.419741 2315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:56.421963 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "9a5954b1a8594a0b9d353b4ffe257fb5"
format_stamp: "Formatted at 2026-08-09 01:56:56 on dist-test-slave-gkw7"
server_key: "11116f43be0cad260e96dc17971f73d6"
server_key_iv: "02bcc86b68e3b72f4744a60e639c0aaa"
server_key_version: "encryptionkey@0"
I20260809 01:56:56.422556 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "9a5954b1a8594a0b9d353b4ffe257fb5"
format_stamp: "Formatted at 2026-08-09 01:56:56 on dist-test-slave-gkw7"
server_key: "11116f43be0cad260e96dc17971f73d6"
server_key_iv: "02bcc86b68e3b72f4744a60e639c0aaa"
server_key_version: "encryptionkey@0"
I20260809 01:56:56.426478 2315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 01:56:56.429011 2334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:56.430323 2315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:56:56.430471 2315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "9a5954b1a8594a0b9d353b4ffe257fb5"
format_stamp: "Formatted at 2026-08-09 01:56:56 on dist-test-slave-gkw7"
server_key: "11116f43be0cad260e96dc17971f73d6"
server_key_iv: "02bcc86b68e3b72f4744a60e639c0aaa"
server_key_version: "encryptionkey@0"
I20260809 01:56:56.430588 2315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:56.449895 2315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:56.452893 2315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:56.453112 2315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:56.460856 2315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41137
I20260809 01:56:56.460875 2415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41137 every 8 connection(s)
I20260809 01:56:56.461898 2315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:56.464699 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 2315
I20260809 01:56:56.464793 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:56.464859 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:56.465167 3952 external_mini_cluster.cc:1428] Setting key 3b3b45699426870c24bcf63dbd3559fc
I20260809 01:56:56.471244 2416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5: Bootstrap starting.
I20260809 01:56:56.473685 2416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:56.474454 2416 log.cc:824] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:56.474834 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:56.466652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:33521 (local address 127.0.0.1:41137)
0809 01:56:56.467142 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:56.467158 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:56.467195 (+ 37us) server_negotiation.cc:407] Connection header received
0809 01:56:56.468036 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:56.468056 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:56.468413 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:56.468759 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:56.469165 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:56.470247 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:56.471180 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:56.471713 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:56.471885 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:56.472897 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:56.472903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:56.472915 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:56.473302 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:56.473367 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:56.473391 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:56.473536 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:56.473929 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 01:56:56.474057 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":135,"threads_started":1}
I20260809 01:56:56.476612 2416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5: No bootstrap required, opened a new log
I20260809 01:56:56.477900 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:56.481391 2416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5954b1a8594a0b9d353b4ffe257fb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41137 } }
I20260809 01:56:56.481684 2416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:56.481801 2416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5954b1a8594a0b9d353b4ffe257fb5, State: Initialized, Role: FOLLOWER
I20260809 01:56:56.482304 2416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5954b1a8594a0b9d353b4ffe257fb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41137 } }
I20260809 01:56:56.482431 2416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:56.482518 2416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:56.482638 2416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:56.483768 2416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5954b1a8594a0b9d353b4ffe257fb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41137 } }
I20260809 01:56:56.484198 2416 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5954b1a8594a0b9d353b4ffe257fb5; no voters:
I20260809 01:56:56.484608 2416 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:56.484754 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:56.480224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:52543 (local address 127.0.0.1:41137)
0809 01:56:56.480374 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:56.480378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:56.480543 (+ 165us) server_negotiation.cc:407] Connection header received
0809 01:56:56.480677 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:56.480679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:56.480720 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:56.480778 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:56.481091 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:56.481648 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:56.482623 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:56.482948 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:56.483065 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:56.484090 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:56.484094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:56.484097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:56.484110 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:56.484138 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:56.484151 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:56.484252 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:56.484541 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 01:56:56.484584 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":53}
I20260809 01:56:56.485040 2433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:56.485338 2433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [term 1 LEADER]: Becoming Leader. State: Replica: 9a5954b1a8594a0b9d353b4ffe257fb5, State: Running, Role: LEADER
I20260809 01:56:56.485764 2433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5954b1a8594a0b9d353b4ffe257fb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41137 } }
I20260809 01:56:56.486065 2416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:56.487000 2433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5954b1a8594a0b9d353b4ffe257fb5. Latest consensus state: current_term: 1 leader_uuid: "9a5954b1a8594a0b9d353b4ffe257fb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5954b1a8594a0b9d353b4ffe257fb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41137 } } }
I20260809 01:56:56.487164 2433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:56.487497 2434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5954b1a8594a0b9d353b4ffe257fb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5954b1a8594a0b9d353b4ffe257fb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41137 } } }
I20260809 01:56:56.487645 2434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:56.487682 2445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:56.491032 2445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:56.497125 2445 catalog_manager.cc:1347] Generated new cluster ID: e5b6522ad4e44ae990980bfb7be6c1a4
I20260809 01:56:56.497227 2445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:56.506294 2445 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:56.507207 2445 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:56.515283 2445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5954b1a8594a0b9d353b4ffe257fb5: Generated new TSK 0
I20260809 01:56:56.515976 2445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:56.524106 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 2315
W20260809 01:56:56.535408 2294 connection.cc:535] client connection to 127.0.0.1:41137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41137): Connection reset by peer (error 104)
2026-08-09T01:56:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:56 dist-test-slave-gkw7 krb5kdc[2463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:56 dist-test-slave-gkw7 krb5kdc[2463](info): set up 2 sockets
Aug 09 01:56:56 dist-test-slave-gkw7 krb5kdc[2463](info): commencing operation
krb5kdc: starting...
W20260809 01:56:57.773290 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:57 dist-test-slave-gkw7 krb5kdc[2463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:57.904107 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:58.021282 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:58.021660 2481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:58.021757 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:58.025894 2481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:58.026016 2481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:58.026057 2481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:58.026096 2481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:58.026132 2481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:58.031697 2481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:58.032998 2481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:58.034168 2481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:58.043112 2496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:58.043438 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:58.043469 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:58.043773 2481 server_base.cc:1034] running on GCE node
I20260809 01:56:58.044433 2481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:58.045709 2481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:58.046876 2481 hybrid_clock.cc:648] HybridClock initialized: now 1786240618046844 us; error 44 us; skew 500 ppm
Aug 09 01:56:58 dist-test-slave-gkw7 krb5kdc[2463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:58.049799 2481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:58.050760 2481 webserver.cc:460] Webserver started at http://127.0.0.1:44013/ using document root <none> and password file <none>
I20260809 01:56:58.051366 2481 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:58.051432 2481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:58.051697 2481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:58.053949 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "f5a606d7b8e546bc92f23716de986d61"
format_stamp: "Formatted at 2026-08-09 01:56:58 on dist-test-slave-gkw7"
server_key: "983744d1a4f89177160f0d9d04af256f"
server_key_iv: "cb91edae6d8f8317cdd2cf2db07ff2fc"
server_key_version: "encryptionkey@0"
I20260809 01:56:58.054554 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "f5a606d7b8e546bc92f23716de986d61"
format_stamp: "Formatted at 2026-08-09 01:56:58 on dist-test-slave-gkw7"
server_key: "983744d1a4f89177160f0d9d04af256f"
server_key_iv: "cb91edae6d8f8317cdd2cf2db07ff2fc"
server_key_version: "encryptionkey@0"
I20260809 01:56:58.058574 2481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:56:58.061223 2505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:58.062567 2481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:56:58.062721 2481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "f5a606d7b8e546bc92f23716de986d61"
format_stamp: "Formatted at 2026-08-09 01:56:58 on dist-test-slave-gkw7"
server_key: "983744d1a4f89177160f0d9d04af256f"
server_key_iv: "cb91edae6d8f8317cdd2cf2db07ff2fc"
server_key_version: "encryptionkey@0"
I20260809 01:56:58.062844 2481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:58.067637 2481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:58.070734 2481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:58.070937 2481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:58.078105 2481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37935
I20260809 01:56:58.078125 2583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37935 every 8 connection(s)
I20260809 01:56:58.079071 2481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:58.081656 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 2481
I20260809 01:56:58.081802 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:58.082157 3952 external_mini_cluster.cc:1428] Setting key b21d6efb8ed2bb5d3c2527b72e850f45
I20260809 01:56:58.082194 2584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:58.087723 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:58.083650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:35065 (local address 127.0.0.1:37935)
0809 01:56:58.084111 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:58.084127 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:58.084163 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:56:58.084976 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:58.084996 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:58.085282 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:58.085656 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:58.086511 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:58.086516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:58.086528 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:58.086538 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:58.086593 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:58.086613 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:58.086773 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:58.086859 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 01:56:58.087028 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":145,"threads_started":1}
I20260809 01:56:58.089263 2584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61: Bootstrap starting.
I20260809 01:56:58.090714 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:58.091813 2584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:58.092582 2584 log.cc:824] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:58.094448 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:58.092775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:33019 (local address 127.0.0.1:37935)
0809 01:56:58.092900 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:58.092904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:58.093118 (+ 214us) server_negotiation.cc:407] Connection header received
0809 01:56:58.093205 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:58.093208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:58.093241 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:58.093328 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:58.094064 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:58.094067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:58.094070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:56:58.094080 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:58.094105 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:56:58.094115 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:58.094185 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:58.094290 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 01:56:58.094324 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 01:56:58.095129 2584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61: No bootstrap required, opened a new log
I20260809 01:56:58.098771 2584 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a606d7b8e546bc92f23716de986d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260809 01:56:58.099006 2584 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:58.099083 2584 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a606d7b8e546bc92f23716de986d61, State: Initialized, Role: FOLLOWER
I20260809 01:56:58.099499 2584 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a606d7b8e546bc92f23716de986d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260809 01:56:58.099618 2584 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:58.099705 2584 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:58.099830 2584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:56:58.100903 2584 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a606d7b8e546bc92f23716de986d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260809 01:56:58.101202 2584 leader_election.cc:302] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5a606d7b8e546bc92f23716de986d61; no voters:
I20260809 01:56:58.101478 2584 leader_election.cc:288] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:58.101639 2607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:58.101928 2607 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [term 1 LEADER]: Becoming Leader. State: Replica: f5a606d7b8e546bc92f23716de986d61, State: Running, Role: LEADER
I20260809 01:56:58.102249 2607 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a606d7b8e546bc92f23716de986d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260809 01:56:58.102778 2584 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:58.104012 2609 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5a606d7b8e546bc92f23716de986d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a606d7b8e546bc92f23716de986d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } } }
I20260809 01:56:58.104140 2609 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:58.104588 2610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a606d7b8e546bc92f23716de986d61. Latest consensus state: current_term: 1 leader_uuid: "f5a606d7b8e546bc92f23716de986d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a606d7b8e546bc92f23716de986d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } } }
I20260809 01:56:58.104702 2610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:58.104580 2622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:58.107851 2622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:58.113969 2622 catalog_manager.cc:1347] Generated new cluster ID: 0975a42e45f94517b6c02ae103b5c3c3
I20260809 01:56:58.114042 2622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:58.133349 2622 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:58.133744 2622 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:58.146617 2622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5a606d7b8e546bc92f23716de986d61: Generated new TSK 0
I20260809 01:56:58.147403 2622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:58.153429 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 2481
2026-08-09T01:56:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1635 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:58 dist-test-slave-gkw7 krb5kdc[2637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:58 dist-test-slave-gkw7 krb5kdc[2637](info): set up 2 sockets
Aug 09 01:56:58 dist-test-slave-gkw7 krb5kdc[2637](info): commencing operation
krb5kdc: starting...
W20260809 01:56:59.392431 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:56:59 dist-test-slave-gkw7 krb5kdc[2637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:56:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:56:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:56:59.520769 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:56:59.634181 2655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:56:59.634500 2655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:56:59.634574 2655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:56:59.638638 2655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:56:59.638728 2655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:56:59.638763 2655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:56:59.638820 2655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:56:59.638875 2655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:56:59.644449 2655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:56:59.645593 2655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:56:59.646659 2655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:56:59.655767 2661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:59.655781 2660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:56:59.655788 2663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:56:59.656122 2655 server_base.cc:1034] running on GCE node
I20260809 01:56:59.656734 2655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:56:59.657943 2655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:56:59.659132 2655 hybrid_clock.cc:648] HybridClock initialized: now 1786240619659100 us; error 45 us; skew 500 ppm
Aug 09 01:56:59 dist-test-slave-gkw7 krb5kdc[2637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:56:59.667763 2655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:56:59.668705 2655 webserver.cc:460] Webserver started at http://127.0.0.1:44353/ using document root <none> and password file <none>
I20260809 01:56:59.669287 2655 fs_manager.cc:359] Metadata directory not provided
I20260809 01:56:59.669402 2655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:56:59.669716 2655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:56:59.671785 2655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "4bbd87ef39694eefb427fc6a95efc168"
format_stamp: "Formatted at 2026-08-09 01:56:59 on dist-test-slave-gkw7"
server_key: "20cff67186d821d20e4b8aa9d5ad6c51"
server_key_iv: "56997a2d6bc65013ec5590723b369812"
server_key_version: "encryptionkey@0"
I20260809 01:56:59.672390 2655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "4bbd87ef39694eefb427fc6a95efc168"
format_stamp: "Formatted at 2026-08-09 01:56:59 on dist-test-slave-gkw7"
server_key: "20cff67186d821d20e4b8aa9d5ad6c51"
server_key_iv: "56997a2d6bc65013ec5590723b369812"
server_key_version: "encryptionkey@0"
I20260809 01:56:59.676117 2655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 01:56:59.678582 2672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:56:59.679774 2655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:56:59.679917 2655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "4bbd87ef39694eefb427fc6a95efc168"
format_stamp: "Formatted at 2026-08-09 01:56:59 on dist-test-slave-gkw7"
server_key: "20cff67186d821d20e4b8aa9d5ad6c51"
server_key_iv: "56997a2d6bc65013ec5590723b369812"
server_key_version: "encryptionkey@0"
I20260809 01:56:59.680030 2655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:56:59.694828 2655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:56:59.700129 2655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:56:59.700347 2655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:56:59.708123 2655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34955
I20260809 01:56:59.708149 2743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34955 every 8 connection(s)
I20260809 01:56:59.709106 2655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:56:59.711894 2747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:56:59.717446 2747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168: Bootstrap starting.
I20260809 01:56:59.719230 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 2655
I20260809 01:56:59.719376 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:56:59.719676 3952 external_mini_cluster.cc:1428] Setting key 0ae5dc5bacf20bf82461a083ff87467b
I20260809 01:56:59.720252 2747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168: Neither blocks nor log segments found. Creating new log.
I20260809 01:56:59.721199 2747 log.cc:824] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168: Log is configured to *not* fsync() on all Append() calls
I20260809 01:56:59.723161 2747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168: No bootstrap required, opened a new log
I20260809 01:56:59.726714 2747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbd87ef39694eefb427fc6a95efc168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } }
I20260809 01:56:59.727028 2747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:56:59.727116 2747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbd87ef39694eefb427fc6a95efc168, State: Initialized, Role: FOLLOWER
I20260809 01:56:59.727564 2747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbd87ef39694eefb427fc6a95efc168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } }
I20260809 01:56:59.727715 2747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:56:59.727804 2747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:56:59.727905 2747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 01:56:59 dist-test-slave-gkw7 krb5kdc[2637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 01:56:59.729135 2747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbd87ef39694eefb427fc6a95efc168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } }
I20260809 01:56:59.729499 2747 leader_election.cc:302] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbd87ef39694eefb427fc6a95efc168; no voters:
I20260809 01:56:59.729939 2747 leader_election.cc:288] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:56:59.730067 2752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:56:59.730314 2752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [term 1 LEADER]: Becoming Leader. State: Replica: 4bbd87ef39694eefb427fc6a95efc168, State: Running, Role: LEADER
I20260809 01:56:59.730758 2752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbd87ef39694eefb427fc6a95efc168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } }
I20260809 01:56:59.731344 2747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:56:59.732613 2755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bbd87ef39694eefb427fc6a95efc168. Latest consensus state: current_term: 1 leader_uuid: "4bbd87ef39694eefb427fc6a95efc168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbd87ef39694eefb427fc6a95efc168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } } }
I20260809 01:56:59.732748 2755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:59.733134 2753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bbd87ef39694eefb427fc6a95efc168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbd87ef39694eefb427fc6a95efc168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34955 } } }
I20260809 01:56:59.733302 2753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168 [sys.catalog]: This master's current role is: LEADER
I20260809 01:56:59.735553 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:59.720868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:37713 (local address 127.0.0.1:34955)
0809 01:56:59.721369 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:59.721385 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:59.721421 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:56:59.722139 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:59.722157 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:59.722506 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:59.722865 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:59.723679 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:59.724655 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:59.725576 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:59.726065 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:59.726237 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:59.729420 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:59.729425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:59.729439 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:59.729466 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:59.731893 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:59.732643 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:59.732649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:59.732655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:59.732803 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:59.733272 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:59.733276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:59.733278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:59.733880 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:59.734127 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:59.734450 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 01:56:59.734677 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
W20260809 01:56:59.736547 2773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:56:59.736626 2773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:56:59.736686 2766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:56:59.737402 2766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:56:59.739090 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:56:59.744036 2766 catalog_manager.cc:1347] Generated new cluster ID: 093d15903fc644f0bb381d2745c03645
I20260809 01:56:59.744100 2766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:56:59.750193 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:56:59.741172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:41307 (local address 127.0.0.1:34955)
0809 01:56:59.741312 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:56:59.741317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:56:59.741878 (+ 561us) server_negotiation.cc:407] Connection header received
0809 01:56:59.742015 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:56:59.742018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:56:59.742068 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:56:59.742155 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:56:59.742888 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:59.743440 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:59.744449 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:56:59.744792 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:56:59.744910 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:56:59.746679 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:56:59.746682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:56:59.746685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:56:59.746708 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:56:59.748278 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:59.748823 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:59.748827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:59.748829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:59.748904 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:56:59.749220 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:56:59.749223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:56:59.749224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:56:59.749426 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:56:59.749529 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 01:56:59.749943 (+ 414us) server_negotiation.cc:299] Negotiation successful
0809 01:56:59.750035 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":57}
I20260809 01:56:59.763347 2766 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:56:59.764626 2766 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:56:59.769959 2766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bbd87ef39694eefb427fc6a95efc168: Generated new TSK 0
I20260809 01:56:59.770736 2766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:56:59.778134 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 2655
W20260809 01:56:59.792203 2635 connection.cc:535] client connection to 127.0.0.1:34955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34955): Connection reset by peer (error 104)
2026-08-09T01:56:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1621 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:56:59 dist-test-slave-gkw7 krb5kdc[2798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:56:59 dist-test-slave-gkw7 krb5kdc[2798](info): set up 2 sockets
Aug 09 01:56:59 dist-test-slave-gkw7 krb5kdc[2798](info): commencing operation
krb5kdc: starting...
W20260809 01:57:00.997440 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:01 dist-test-slave-gkw7 krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:57:01.126547 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:01.241665 2814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:01.241967 2814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:01.242012 2814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:01.245963 2814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:01.246028 2814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:01.246047 2814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:01.246064 2814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:01.246079 2814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:01.251403 2814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:01.252537 2814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:01.253595 2814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:01.262737 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:01.262739 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:01.262741 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:01.262791 2814 server_base.cc:1034] running on GCE node
I20260809 01:57:01.263700 2814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:01.264791 2814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:01.265961 2814 hybrid_clock.cc:648] HybridClock initialized: now 1786240621265942 us; error 31 us; skew 500 ppm
Aug 09 01:57:01 dist-test-slave-gkw7 krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:01.268697 2814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:57:01.269717 2814 webserver.cc:460] Webserver started at http://127.0.0.1:40001/ using document root <none> and password file <none>
I20260809 01:57:01.270291 2814 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:01.270361 2814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:01.270619 2814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:01.272962 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "313ef304ee1041dcb1e2cd064074a634"
format_stamp: "Formatted at 2026-08-09 01:57:01 on dist-test-slave-gkw7"
server_key: "48723c5d142c5bd3f0fac398bd293546"
server_key_iv: "d534c1f950406b25844a93889a1a8345"
server_key_version: "encryptionkey@0"
I20260809 01:57:01.273813 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "313ef304ee1041dcb1e2cd064074a634"
format_stamp: "Formatted at 2026-08-09 01:57:01 on dist-test-slave-gkw7"
server_key: "48723c5d142c5bd3f0fac398bd293546"
server_key_iv: "d534c1f950406b25844a93889a1a8345"
server_key_version: "encryptionkey@0"
I20260809 01:57:01.279361 2814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 01:57:01.282557 2837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:01.283840 2814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:57:01.283980 2814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "313ef304ee1041dcb1e2cd064074a634"
format_stamp: "Formatted at 2026-08-09 01:57:01 on dist-test-slave-gkw7"
server_key: "48723c5d142c5bd3f0fac398bd293546"
server_key_iv: "d534c1f950406b25844a93889a1a8345"
server_key_version: "encryptionkey@0"
I20260809 01:57:01.284097 2814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:01.312985 2814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:01.316211 2814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:01.316447 2814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:01.323827 2814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45883
I20260809 01:57:01.323848 2918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45883 every 8 connection(s)
I20260809 01:57:01.324880 2814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:01.327702 2919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:01.333204 2919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634: Bootstrap starting.
I20260809 01:57:01.334430 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 2814
I20260809 01:57:01.334573 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:01.334883 3952 external_mini_cluster.cc:1428] Setting key 625816773e0671f9dad0e9b297031f6c
I20260809 01:57:01.336241 2919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:01.337014 2919 log.cc:824] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:01.339012 2919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634: No bootstrap required, opened a new log
I20260809 01:57:01.342643 2919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313ef304ee1041dcb1e2cd064074a634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } }
I20260809 01:57:01.342893 2919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:57:01.343012 2919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313ef304ee1041dcb1e2cd064074a634, State: Initialized, Role: FOLLOWER
I20260809 01:57:01.343535 2919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313ef304ee1041dcb1e2cd064074a634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } }
I20260809 01:57:01.343665 2919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:57:01.343748 2919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:57:01.343865 2919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:01.344050 2925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:01.336079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:60889 (local address 127.0.0.1:45883)
0809 01:57:01.336556 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:01.336572 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:01.336607 (+ 35us) server_negotiation.cc:407] Connection header received
0809 01:57:01.337332 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:01.337352 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:01.337769 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:01.338122 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:01.338552 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:01.339597 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:01.340511 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:01.340999 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:01.341172 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:01.342170 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:01.342175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:01.342189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:01.342590 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:01.342647 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:01.342670 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:01.342816 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:01.342973 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 01:57:01.343119 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":136,"threads_started":1}
I20260809 01:57:01.344841 2919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313ef304ee1041dcb1e2cd064074a634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } }
I20260809 01:57:01.345188 2919 leader_election.cc:302] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 313ef304ee1041dcb1e2cd064074a634; no voters:
I20260809 01:57:01.345575 2919 leader_election.cc:288] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:57:01.345652 2928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:01.345923 2928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [term 1 LEADER]: Becoming Leader. State: Replica: 313ef304ee1041dcb1e2cd064074a634, State: Running, Role: LEADER
I20260809 01:57:01.346304 2928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313ef304ee1041dcb1e2cd064074a634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } }
I20260809 01:57:01.347000 2919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:01.347339 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:01.347589 2928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313ef304ee1041dcb1e2cd064074a634. Latest consensus state: current_term: 1 leader_uuid: "313ef304ee1041dcb1e2cd064074a634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313ef304ee1041dcb1e2cd064074a634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } } }
I20260809 01:57:01.347700 2928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:01.348083 2934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:01.348294 2929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313ef304ee1041dcb1e2cd064074a634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313ef304ee1041dcb1e2cd064074a634" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45883 } } }
I20260809 01:57:01.348461 2929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:01.351431 2934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:01.356344 2925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:01.351218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:45249 (local address 127.0.0.1:45883)
0809 01:57:01.351317 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:01.351322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:01.351986 (+ 664us) server_negotiation.cc:407] Connection header received
0809 01:57:01.352194 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:01.352198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:01.352248 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:01.352337 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:01.352617 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:01.353140 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:01.354277 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:01.354649 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:01.354767 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:01.355776 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:01.355780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:01.355783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:01.355796 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:01.355826 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:01.355840 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:01.355925 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:01.356157 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 01:57:01.356195 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 01:57:01.357806 2934 catalog_manager.cc:1347] Generated new cluster ID: b7117b4574384a7dbfc461c3bc7847d0
I20260809 01:57:01.357874 2934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:01.370836 2934 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:01.371834 2934 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:01.380563 2934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 313ef304ee1041dcb1e2cd064074a634: Generated new TSK 0
I20260809 01:57:01.381445 2934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:57:01.390722 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 2814
W20260809 01:57:01.405330 2792 connection.cc:535] client connection to 127.0.0.1:45883 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45883): Connection reset by peer (error 104)
2026-08-09T01:57:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:01 dist-test-slave-gkw7 krb5kdc[2963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:01 dist-test-slave-gkw7 krb5kdc[2963](info): set up 2 sockets
Aug 09 01:57:01 dist-test-slave-gkw7 krb5kdc[2963](info): commencing operation
krb5kdc: starting...
W20260809 01:57:02.710457 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:02 dist-test-slave-gkw7 krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 01:57:02.836604 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:02.950160 2984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:02.950491 2984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:02.950577 2984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:02.954550 2984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:02.954649 2984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:02.954690 2984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:02.954751 2984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:02.954792 2984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:02.960089 2984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:02.961221 2984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:02.962363 2984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:02.971338 2989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:02.971344 2990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:02.971410 2984 server_base.cc:1034] running on GCE node
W20260809 01:57:02.971338 2992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:02.972075 2984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:02.973184 2984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:02.974375 2984 hybrid_clock.cc:648] HybridClock initialized: now 1786240622974342 us; error 46 us; skew 500 ppm
Aug 09 01:57:02 dist-test-slave-gkw7 krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:02.977108 2984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 01:57:02.978067 2984 webserver.cc:460] Webserver started at http://127.0.0.1:36399/ using document root <none> and password file <none>
I20260809 01:57:02.978591 2984 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:02.978650 2984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:02.978812 2984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:02.980804 2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "d94f9847bf3240baab1c5bc0fbf943e0"
format_stamp: "Formatted at 2026-08-09 01:57:02 on dist-test-slave-gkw7"
server_key: "886577456659e8716557d8aa4d565b97"
server_key_iv: "9b04321e0d8c0810d5cd24cca0bc5ad1"
server_key_version: "encryptionkey@0"
I20260809 01:57:02.981324 2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "d94f9847bf3240baab1c5bc0fbf943e0"
format_stamp: "Formatted at 2026-08-09 01:57:02 on dist-test-slave-gkw7"
server_key: "886577456659e8716557d8aa4d565b97"
server_key_iv: "9b04321e0d8c0810d5cd24cca0bc5ad1"
server_key_version: "encryptionkey@0"
I20260809 01:57:02.984982 2984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 01:57:02.987349 3005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:02.988555 2984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:57:02.988688 2984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "d94f9847bf3240baab1c5bc0fbf943e0"
format_stamp: "Formatted at 2026-08-09 01:57:02 on dist-test-slave-gkw7"
server_key: "886577456659e8716557d8aa4d565b97"
server_key_iv: "9b04321e0d8c0810d5cd24cca0bc5ad1"
server_key_version: "encryptionkey@0"
I20260809 01:57:02.988790 2984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:02.993405 2984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:02.996196 2984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:02.996404 2984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:03.003306 2984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39379
I20260809 01:57:03.003382 3080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39379 every 8 connection(s)
I20260809 01:57:03.004284 2984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:03.007261 3081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:03.012866 3081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0: Bootstrap starting.
I20260809 01:57:03.013517 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 2984
I20260809 01:57:03.013725 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:03.014012 3952 external_mini_cluster.cc:1428] Setting key a24f5d6f4c73c25b4f7df280677c71bd
I20260809 01:57:03.015791 3081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:03.016537 3081 log.cc:824] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:03.018914 3081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0: No bootstrap required, opened a new log
I20260809 01:57:03.019614 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:03.015251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:54889 (local address 127.0.0.1:39379)
0809 01:57:03.015724 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:03.015739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:03.015775 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:57:03.016472 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:03.016490 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:03.016837 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:03.017250 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:03.018140 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:03.018145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:03.018157 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:03.018167 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:03.018219 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:03.018239 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:03.018330 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:03.018488 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 01:57:03.018678 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":137,"threads_started":1}
I20260809 01:57:03.022893 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:03.023708 3081 raft_consensus.cc:348] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94f9847bf3240baab1c5bc0fbf943e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39379 } }
I20260809 01:57:03.024017 3081 raft_consensus.cc:374] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:57:03.024080 3081 raft_consensus.cc:729] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94f9847bf3240baab1c5bc0fbf943e0, State: Initialized, Role: FOLLOWER
I20260809 01:57:03.024632 3081 consensus_queue.cc:260] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94f9847bf3240baab1c5bc0fbf943e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39379 } }
I20260809 01:57:03.024811 3081 raft_consensus.cc:388] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:57:03.024889 3081 raft_consensus.cc:482] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:57:03.025028 3081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:03.026407 3081 raft_consensus.cc:504] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94f9847bf3240baab1c5bc0fbf943e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39379 } }
I20260809 01:57:03.026793 3081 leader_election.cc:302] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d94f9847bf3240baab1c5bc0fbf943e0; no voters:
I20260809 01:57:03.027092 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:03.025216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.125:35997 (local address 127.0.0.1:39379)
0809 01:57:03.025366 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:03.025371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:03.025607 (+ 236us) server_negotiation.cc:407] Connection header received
0809 01:57:03.025739 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:03.025741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:03.025774 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:03.025843 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:03.026620 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:03.026623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:03.026626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:03.026638 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:03.026672 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:03.026683 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:03.026753 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:03.026870 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 01:57:03.026904 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 01:57:03.027194 3081 leader_election.cc:288] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:57:03.027259 3094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:03.027498 3094 raft_consensus.cc:686] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [term 1 LEADER]: Becoming Leader. State: Replica: d94f9847bf3240baab1c5bc0fbf943e0, State: Running, Role: LEADER
I20260809 01:57:03.027803 3094 consensus_queue.cc:237] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94f9847bf3240baab1c5bc0fbf943e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39379 } }
I20260809 01:57:03.028523 3081 sys_catalog.cc:564] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:03.029783 3096 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d94f9847bf3240baab1c5bc0fbf943e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94f9847bf3240baab1c5bc0fbf943e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39379 } } }
I20260809 01:57:03.029923 3096 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:03.030493 3110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:03.030267 3095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d94f9847bf3240baab1c5bc0fbf943e0. Latest consensus state: current_term: 1 leader_uuid: "d94f9847bf3240baab1c5bc0fbf943e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94f9847bf3240baab1c5bc0fbf943e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39379 } } }
I20260809 01:57:03.031530 3095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:03.033842 3110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:03.039379 3110 catalog_manager.cc:1347] Generated new cluster ID: 92db42636e5d4d33850c4bede7e51df0
I20260809 01:57:03.039449 3110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:03.051772 3110 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:03.052143 3110 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:03.076938 3110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d94f9847bf3240baab1c5bc0fbf943e0: Generated new TSK 0
I20260809 01:57:03.077757 3110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:57:03.084704 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 2984
2026-08-09T01:57:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1689 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:03 dist-test-slave-gkw7 krb5kdc[3128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:03 dist-test-slave-gkw7 krb5kdc[3128](info): set up 2 sockets
Aug 09 01:57:03 dist-test-slave-gkw7 krb5kdc[3128](info): commencing operation
krb5kdc: starting...
W20260809 01:57:04.334559 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:04 dist-test-slave-gkw7 krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:57:04.460616 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:43543
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:43543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:04.570724 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:04.571058 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:04.571134 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:04.575253 3152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:04.575346 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:04.575388 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:04.575448 3152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:04.575487 3152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:04.580649 3152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:43543
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:43543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:04.581691 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:04.582708 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:04.591421 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:04.591566 3152 server_base.cc:1034] running on GCE node
W20260809 01:57:04.591423 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:04.591444 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:04.592278 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:04.593451 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:04.594708 3152 hybrid_clock.cc:648] HybridClock initialized: now 1786240624594677 us; error 41 us; skew 500 ppm
Aug 09 01:57:04 dist-test-slave-gkw7 krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:04.597754 3152 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:57:04.598686 3152 webserver.cc:460] Webserver started at http://127.3.220.62:42771/ using document root <none> and password file <none>
I20260809 01:57:04.599284 3152 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:04.599387 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:04.599649 3152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:04.602046 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "11762179912741888096d94248aebe89"
format_stamp: "Formatted at 2026-08-09 01:57:04 on dist-test-slave-gkw7"
server_key: "16969b1299c5fbe121d8ab8c82d6b2e1"
server_key_iv: "88599efbd78047110931118dfa8cc8a0"
server_key_version: "encryptionkey@0"
I20260809 01:57:04.602828 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "11762179912741888096d94248aebe89"
format_stamp: "Formatted at 2026-08-09 01:57:04 on dist-test-slave-gkw7"
server_key: "16969b1299c5fbe121d8ab8c82d6b2e1"
server_key_iv: "88599efbd78047110931118dfa8cc8a0"
server_key_version: "encryptionkey@0"
I20260809 01:57:04.606874 3152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:57:04.609493 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:04.610913 3152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 01:57:04.611059 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "11762179912741888096d94248aebe89"
format_stamp: "Formatted at 2026-08-09 01:57:04 on dist-test-slave-gkw7"
server_key: "16969b1299c5fbe121d8ab8c82d6b2e1"
server_key_iv: "88599efbd78047110931118dfa8cc8a0"
server_key_version: "encryptionkey@0"
I20260809 01:57:04.611176 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:04.629822 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:04.633147 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:04.633369 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:04.640803 3152 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:43543
I20260809 01:57:04.640833 3249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:43543 every 8 connection(s)
I20260809 01:57:04.641752 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:04.644508 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:04.647066 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 3152
I20260809 01:57:04.647233 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:04.647503 3952 external_mini_cluster.cc:1428] Setting key 3cbcb138b3efd1cb0bf281a6a8fc98cb
I20260809 01:57:04.650583 3251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89: Bootstrap starting.
I20260809 01:57:04.652789 3251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:04.653595 3251 log.cc:824] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:04.655956 3251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89: No bootstrap required, opened a new log
Aug 09 01:57:04 dist-test-slave-gkw7 krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:57:04.660586 3251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11762179912741888096d94248aebe89" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 43543 } }
I20260809 01:57:04.660887 3251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:57:04.660959 3251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11762179912741888096d94248aebe89, State: Initialized, Role: FOLLOWER
I20260809 01:57:04.661510 3251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11762179912741888096d94248aebe89" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 43543 } }
I20260809 01:57:04.661691 3251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:57:04.661759 3251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:57:04.661875 3251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:04.663007 3251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11762179912741888096d94248aebe89" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 43543 } }
I20260809 01:57:04.663384 3251 leader_election.cc:302] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11762179912741888096d94248aebe89; no voters:
I20260809 01:57:04.663652 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:04.648900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40219 (local address 127.3.220.62:43543)
0809 01:57:04.649496 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:04.649518 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:04.649641 (+ 123us) server_negotiation.cc:407] Connection header received
0809 01:57:04.650303 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:04.650320 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:04.650685 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:04.651081 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:04.652045 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:04.653144 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:04.654282 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:04.654946 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:04.655110 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:04.658108 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:04.658115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:04.658131 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:04.658161 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:04.660204 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:04.660716 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:04.660722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:04.660728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:04.660880 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:04.661283 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:04.661287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:04.661288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:04.661829 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:04.662082 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:04.662390 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 01:57:04.662617 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":132,"threads_started":1}
I20260809 01:57:04.663736 3251 leader_election.cc:288] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:57:04.663901 3259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:04.664124 3259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [term 1 LEADER]: Becoming Leader. State: Replica: 11762179912741888096d94248aebe89, State: Running, Role: LEADER
I20260809 01:57:04.664465 3259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11762179912741888096d94248aebe89" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 43543 } }
I20260809 01:57:04.664956 3251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:04.665766 3263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11762179912741888096d94248aebe89. Latest consensus state: current_term: 1 leader_uuid: "11762179912741888096d94248aebe89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11762179912741888096d94248aebe89" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 43543 } } }
I20260809 01:57:04.665940 3263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:04.666349 3272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:04.667091 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:04.667431 3262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11762179912741888096d94248aebe89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11762179912741888096d94248aebe89" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 43543 } } }
I20260809 01:57:04.667516 3262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:04.669665 3272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:04.675732 3272 catalog_manager.cc:1347] Generated new cluster ID: a720affadecf4074a870395a5bd485aa
I20260809 01:57:04.675791 3272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:04.683710 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:04.669461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34667 (local address 127.3.220.62:43543)
0809 01:57:04.669591 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:04.669597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:04.671691 (+ 2094us) server_negotiation.cc:407] Connection header received
0809 01:57:04.671781 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:04.671784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:04.671846 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:04.671936 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:04.672761 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:04.673633 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:04.674706 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:04.675076 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:04.675186 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:04.680100 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:04.680104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:04.680107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:04.680133 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:04.681871 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:04.682384 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:04.682387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:04.682389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:04.682476 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:04.682821 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:04.682824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:04.682826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:04.683037 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:04.683184 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:04.683457 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 01:57:04.683556 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 01:57:04.687071 3272 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:04.687940 3272 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:04.696103 3272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11762179912741888096d94248aebe89: Generated new TSK 0
I20260809 01:57:04.696872 3272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:57:04.704799 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 3152
W20260809 01:57:04.717118 3126 connection.cc:535] client connection to 127.3.220.62:43543 recv error: Network error: failed to read from TLS socket (remote: 127.3.220.62:43543): Connection reset by peer (error 104)
2026-08-09T01:57:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1621 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:04 dist-test-slave-gkw7 krb5kdc[3302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:04 dist-test-slave-gkw7 krb5kdc[3302](info): set up 2 sockets
Aug 09 01:57:04 dist-test-slave-gkw7 krb5kdc[3302](info): commencing operation
krb5kdc: starting...
W20260809 01:57:05.939714 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:06 dist-test-slave-gkw7 krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:57:06.068730 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:40303
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:46803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:06.183013 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:06.183341 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:06.183420 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:06.187505 3326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:06.187587 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:06.187625 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:06.187685 3326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:06.187721 3326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:06.192929 3326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:46803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:40303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:06.194111 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:06.195188 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:06.204035 3337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:06.204012 3333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:06.204015 3335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:06.204380 3326 server_base.cc:1034] running on GCE node
I20260809 01:57:06.205049 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:06.206259 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:06.207465 3326 hybrid_clock.cc:648] HybridClock initialized: now 1786240626207437 us; error 42 us; skew 500 ppm
Aug 09 01:57:06 dist-test-slave-gkw7 krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:06.210394 3326 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:57:06.211335 3326 webserver.cc:460] Webserver started at http://127.3.220.62:40517/ using document root <none> and password file <none>
I20260809 01:57:06.211911 3326 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:06.212005 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:06.212245 3326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:06.214507 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "7c5f602768b845fdb8082a238a4282ed"
format_stamp: "Formatted at 2026-08-09 01:57:06 on dist-test-slave-gkw7"
server_key: "a703b57191814e2479e211d1141a1f91"
server_key_iv: "46b134851204bc973b2947e783daa8c7"
server_key_version: "encryptionkey@0"
I20260809 01:57:06.215092 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "7c5f602768b845fdb8082a238a4282ed"
format_stamp: "Formatted at 2026-08-09 01:57:06 on dist-test-slave-gkw7"
server_key: "a703b57191814e2479e211d1141a1f91"
server_key_iv: "46b134851204bc973b2947e783daa8c7"
server_key_version: "encryptionkey@0"
I20260809 01:57:06.218786 3326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 01:57:06.221149 3350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:06.222406 3326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:57:06.222548 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "7c5f602768b845fdb8082a238a4282ed"
format_stamp: "Formatted at 2026-08-09 01:57:06 on dist-test-slave-gkw7"
server_key: "a703b57191814e2479e211d1141a1f91"
server_key_iv: "46b134851204bc973b2947e783daa8c7"
server_key_version: "encryptionkey@0"
I20260809 01:57:06.222658 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:06.242430 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:06.245879 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:06.246094 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:06.253368 3326 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:40303
I20260809 01:57:06.253403 3437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:40303 every 8 connection(s)
I20260809 01:57:06.254384 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:06.255605 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 3326
I20260809 01:57:06.255772 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:06.256095 3952 external_mini_cluster.cc:1428] Setting key 8d299f5bbbab640e53c83bfb3e3035bb
I20260809 01:57:06.257637 3438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:06.263206 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed: Bootstrap starting.
I20260809 01:57:06.265018 3447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:06.257513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42745 (local address 127.3.220.62:40303)
0809 01:57:06.257994 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:06.258009 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:06.258045 (+ 36us) server_negotiation.cc:407] Connection header received
0809 01:57:06.258672 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:06.258687 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:06.259023 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:06.259357 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:06.259759 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:06.260752 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:06.261733 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:06.262227 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:06.262405 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:06.263383 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:06.263389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:06.263406 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:06.263797 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:06.263860 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:06.263884 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:06.264029 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:06.264206 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 01:57:06.264340 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":158,"threads_started":1}
I20260809 01:57:06.265626 3438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:06.266403 3438 log.cc:824] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:06.267964 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:06.268517 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed: No bootstrap required, opened a new log
I20260809 01:57:06.272843 3438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5f602768b845fdb8082a238a4282ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40303 } }
I20260809 01:57:06.273061 3438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:57:06.273140 3438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5f602768b845fdb8082a238a4282ed, State: Initialized, Role: FOLLOWER
I20260809 01:57:06.273648 3438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5f602768b845fdb8082a238a4282ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40303 } }
I20260809 01:57:06.273804 3438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:57:06.273883 3438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:57:06.273990 3438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:06.274420 3447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:06.270093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57471 (local address 127.3.220.62:40303)
0809 01:57:06.270216 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:06.270221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:06.270378 (+ 157us) server_negotiation.cc:407] Connection header received
0809 01:57:06.270459 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:06.270462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:06.270511 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:06.270609 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:06.270869 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:06.271668 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:06.272566 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:06.272786 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:06.272908 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:06.273795 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:06.273799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:06.273801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:06.273814 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:06.273842 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:06.273854 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:06.273968 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:06.274221 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 01:57:06.274254 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 01:57:06.275069 3438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5f602768b845fdb8082a238a4282ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40303 } }
I20260809 01:57:06.275414 3438 leader_election.cc:302] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c5f602768b845fdb8082a238a4282ed; no voters:
I20260809 01:57:06.275714 3438 leader_election.cc:288] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:57:06.275841 3456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:06.276108 3456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [term 1 LEADER]: Becoming Leader. State: Replica: 7c5f602768b845fdb8082a238a4282ed, State: Running, Role: LEADER
I20260809 01:57:06.276440 3456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5f602768b845fdb8082a238a4282ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40303 } }
I20260809 01:57:06.276919 3438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:06.278187 3458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c5f602768b845fdb8082a238a4282ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5f602768b845fdb8082a238a4282ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40303 } } }
I20260809 01:57:06.278178 3459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c5f602768b845fdb8082a238a4282ed. Latest consensus state: current_term: 1 leader_uuid: "7c5f602768b845fdb8082a238a4282ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5f602768b845fdb8082a238a4282ed" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 40303 } } }
I20260809 01:57:06.278332 3459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:06.278332 3458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:06.278759 3471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:06.281440 3471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:06.286664 3471 catalog_manager.cc:1347] Generated new cluster ID: bb6f882d15c5433999f60a4778bce149
I20260809 01:57:06.286736 3471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:06.301504 3471 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:06.302975 3471 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:06.315554 3471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c5f602768b845fdb8082a238a4282ed: Generated new TSK 0
I20260809 01:57:06.316342 3471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:57:06.325609 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 3326
W20260809 01:57:06.341737 3300 connection.cc:535] client connection to 127.3.220.62:40303 recv error: Network error: failed to read from TLS socket (remote: 127.3.220.62:40303): Connection reset by peer (error 104)
2026-08-09T01:57:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1624 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:06 dist-test-slave-gkw7 krb5kdc[3487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:06 dist-test-slave-gkw7 krb5kdc[3487](info): set up 2 sockets
Aug 09 01:57:06 dist-test-slave-gkw7 krb5kdc[3487](info): commencing operation
krb5kdc: starting...
W20260809 01:57:07.562243 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:07 dist-test-slave-gkw7 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:57:07.690223 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:33071
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:07.803575 3514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:07.803858 3514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:07.803934 3514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:07.807888 3514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:07.807973 3514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:07.808022 3514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:07.808079 3514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:07.808120 3514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:07.813218 3514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:33071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:07.814275 3514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:07.815306 3514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:07.824061 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:07.824175 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:07.824201 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:07.824342 3514 server_base.cc:1034] running on GCE node
I20260809 01:57:07.824958 3514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:07.826120 3514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:07.827313 3514 hybrid_clock.cc:648] HybridClock initialized: now 1786240627827300 us; error 32 us; skew 500 ppm
Aug 09 01:57:07 dist-test-slave-gkw7 krb5kdc[3487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:07.830070 3514 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:57:07.831032 3514 webserver.cc:460] Webserver started at http://127.3.220.62:39289/ using document root <none> and password file <none>
I20260809 01:57:07.831600 3514 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:07.831694 3514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:07.831947 3514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:07.834091 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "6f4d29f2463a4708b6ec7dd0f6963505"
format_stamp: "Formatted at 2026-08-09 01:57:07 on dist-test-slave-gkw7"
server_key: "2e44c67d6a4ac074315a9ddb32b13fc3"
server_key_iv: "544ae680c9d3465b8a12b04c2d2ceda1"
server_key_version: "encryptionkey@0"
I20260809 01:57:07.834661 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "6f4d29f2463a4708b6ec7dd0f6963505"
format_stamp: "Formatted at 2026-08-09 01:57:07 on dist-test-slave-gkw7"
server_key: "2e44c67d6a4ac074315a9ddb32b13fc3"
server_key_iv: "544ae680c9d3465b8a12b04c2d2ceda1"
server_key_version: "encryptionkey@0"
I20260809 01:57:07.838413 3514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 01:57:07.840905 3534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:07.842140 3514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:57:07.842294 3514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "6f4d29f2463a4708b6ec7dd0f6963505"
format_stamp: "Formatted at 2026-08-09 01:57:07 on dist-test-slave-gkw7"
server_key: "2e44c67d6a4ac074315a9ddb32b13fc3"
server_key_iv: "544ae680c9d3465b8a12b04c2d2ceda1"
server_key_version: "encryptionkey@0"
I20260809 01:57:07.842404 3514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:07.846967 3514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:07.849781 3514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:07.849982 3514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:07.857167 3514 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:33071
I20260809 01:57:07.857194 3606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:33071 every 8 connection(s)
I20260809 01:57:07.858117 3514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:07.860849 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:07.866307 3607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505: Bootstrap starting.
I20260809 01:57:07.866752 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 3514
I20260809 01:57:07.866902 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:07.867179 3952 external_mini_cluster.cc:1428] Setting key 046eec574060ea5e1b70b7f1189b15e9
I20260809 01:57:07.869170 3607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:07.870131 3607 log.cc:824] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:07.872193 3607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505: No bootstrap required, opened a new log
I20260809 01:57:07.872848 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:07.868634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59911 (local address 127.3.220.62:33071)
0809 01:57:07.869078 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:07.869098 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:07.869144 (+ 46us) server_negotiation.cc:407] Connection header received
0809 01:57:07.870046 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:07.870069 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:07.870440 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:07.870818 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:07.871624 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:07.871629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:07.871640 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:07.871651 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:07.871703 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:07.871722 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:07.871840 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:07.871968 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 01:57:07.872130 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260809 01:57:07.875721 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:07.876350 3607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4d29f2463a4708b6ec7dd0f6963505" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33071 } }
I20260809 01:57:07.876577 3607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:57:07.876657 3607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4d29f2463a4708b6ec7dd0f6963505, State: Initialized, Role: FOLLOWER
I20260809 01:57:07.877139 3607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4d29f2463a4708b6ec7dd0f6963505" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33071 } }
I20260809 01:57:07.877305 3607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:57:07.877394 3607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:57:07.877494 3607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:07.878615 3607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4d29f2463a4708b6ec7dd0f6963505" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33071 } }
I20260809 01:57:07.879042 3607 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f4d29f2463a4708b6ec7dd0f6963505; no voters:
I20260809 01:57:07.879439 3607 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:57:07.879540 3621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:07.879645 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:07.877915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57481 (local address 127.3.220.62:33071)
0809 01:57:07.878038 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:07.878043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:07.878271 (+ 228us) server_negotiation.cc:407] Connection header received
0809 01:57:07.878368 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:07.878371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:07.878406 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:07.878491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:07.879177 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:07.879180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:07.879183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:07.879195 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:07.879223 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:07.879236 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:07.879342 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:07.879401 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 01:57:07.879438 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 01:57:07.879840 3621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [term 1 LEADER]: Becoming Leader. State: Replica: 6f4d29f2463a4708b6ec7dd0f6963505, State: Running, Role: LEADER
I20260809 01:57:07.880194 3621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4d29f2463a4708b6ec7dd0f6963505" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33071 } }
I20260809 01:57:07.880857 3607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:07.881378 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f4d29f2463a4708b6ec7dd0f6963505. Latest consensus state: current_term: 1 leader_uuid: "6f4d29f2463a4708b6ec7dd0f6963505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4d29f2463a4708b6ec7dd0f6963505" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33071 } } }
I20260809 01:57:07.881574 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:07.881850 3622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f4d29f2463a4708b6ec7dd0f6963505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4d29f2463a4708b6ec7dd0f6963505" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 33071 } } }
I20260809 01:57:07.882007 3622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:07.882107 3630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:07.885922 3630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:07.891260 3630 catalog_manager.cc:1347] Generated new cluster ID: 9385bba98ff84e9c990fcf03b89e201f
I20260809 01:57:07.891379 3630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:07.917160 3630 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:07.917508 3630 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:07.929132 3630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f4d29f2463a4708b6ec7dd0f6963505: Generated new TSK 0
I20260809 01:57:07.929929 3630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:57:07.943619 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 3514
2026-08-09T01:57:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1616 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:08 dist-test-slave-gkw7 krb5kdc[3654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:08 dist-test-slave-gkw7 krb5kdc[3654](info): set up 2 sockets
Aug 09 01:57:08 dist-test-slave-gkw7 krb5kdc[3654](info): commencing operation
krb5kdc: starting...
W20260809 01:57:09.208880 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:09 dist-test-slave-gkw7 krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:57:09.344415 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44357
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:09.460413 3670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:09.460741 3670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:09.460817 3670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:09.464850 3670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:09.464956 3670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:09.464993 3670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:09.465030 3670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:09.465065 3670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:09.470377 3670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:44357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:44357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:09.471480 3670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:09.472527 3670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:09.481088 3682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:09.481109 3683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:09.481217 3686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:09.481387 3670 server_base.cc:1034] running on GCE node
I20260809 01:57:09.482085 3670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:09.483215 3670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:09.484423 3670 hybrid_clock.cc:648] HybridClock initialized: now 1786240629484405 us; error 30 us; skew 500 ppm
Aug 09 01:57:09 dist-test-slave-gkw7 krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:09.487298 3670 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:57:09.488176 3670 webserver.cc:460] Webserver started at http://127.3.220.62:33491/ using document root <none> and password file <none>
I20260809 01:57:09.488705 3670 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:09.488765 3670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:09.488940 3670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:09.491173 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "53dd893946554f8ca99d7051136c2440"
format_stamp: "Formatted at 2026-08-09 01:57:09 on dist-test-slave-gkw7"
server_key: "53e00b7ac60b943598bf7223eb116473"
server_key_iv: "57afc3c39f6ecf37ddb1c40488a3c842"
server_key_version: "encryptionkey@0"
I20260809 01:57:09.491704 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "53dd893946554f8ca99d7051136c2440"
format_stamp: "Formatted at 2026-08-09 01:57:09 on dist-test-slave-gkw7"
server_key: "53e00b7ac60b943598bf7223eb116473"
server_key_iv: "57afc3c39f6ecf37ddb1c40488a3c842"
server_key_version: "encryptionkey@0"
I20260809 01:57:09.495440 3670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 01:57:09.497861 3696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:09.499111 3670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:57:09.499215 3670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "53dd893946554f8ca99d7051136c2440"
format_stamp: "Formatted at 2026-08-09 01:57:09 on dist-test-slave-gkw7"
server_key: "53e00b7ac60b943598bf7223eb116473"
server_key_iv: "57afc3c39f6ecf37ddb1c40488a3c842"
server_key_version: "encryptionkey@0"
I20260809 01:57:09.499300 3670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:09.516222 3670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:09.519694 3670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:09.519871 3670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:09.527448 3670 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:44357
I20260809 01:57:09.527529 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:44357 every 8 connection(s)
I20260809 01:57:09.528527 3670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:09.531507 3771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:09.532020 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 3670
I20260809 01:57:09.532164 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:09.532462 3952 external_mini_cluster.cc:1428] Setting key 79ca2150ec21be1fb2955809c13b4e59
I20260809 01:57:09.538673 3771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440: Bootstrap starting.
I20260809 01:57:09.541254 3771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:09.542110 3771 log.cc:824] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:57:09 dist-test-slave-gkw7 krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:57:09.544423 3771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440: No bootstrap required, opened a new log
I20260809 01:57:09.547616 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:09.534005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51061 (local address 127.3.220.62:44357)
0809 01:57:09.534440 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:09.534460 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:09.534506 (+ 46us) server_negotiation.cc:407] Connection header received
0809 01:57:09.535342 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:09.535368 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:09.535816 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:09.536196 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:09.536919 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:09.538461 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:09.539367 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:09.540074 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:09.540237 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:09.542950 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:09.542959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:09.542977 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:09.543014 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:09.544746 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:09.545231 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:09.545236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:09.545240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:09.545360 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:09.545674 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:09.545677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:09.545678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:09.546072 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:09.546218 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:09.546574 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 01:57:09.546866 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":107,"threads_started":1}
I20260809 01:57:09.548583 3771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd893946554f8ca99d7051136c2440" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44357 } }
I20260809 01:57:09.548888 3771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:57:09.548949 3771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53dd893946554f8ca99d7051136c2440, State: Initialized, Role: FOLLOWER
I20260809 01:57:09.549408 3771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd893946554f8ca99d7051136c2440" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44357 } }
I20260809 01:57:09.549619 3771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:57:09.549731 3771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:57:09.549851 3771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:09.550741 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:09.551061 3771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd893946554f8ca99d7051136c2440" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44357 } }
I20260809 01:57:09.551438 3771 leader_election.cc:302] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53dd893946554f8ca99d7051136c2440; no voters:
I20260809 01:57:09.551820 3771 leader_election.cc:288] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:57:09.551927 3779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:09.552174 3779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [term 1 LEADER]: Becoming Leader. State: Replica: 53dd893946554f8ca99d7051136c2440, State: Running, Role: LEADER
I20260809 01:57:09.552541 3779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd893946554f8ca99d7051136c2440" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44357 } }
I20260809 01:57:09.553085 3771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:09.554222 3783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53dd893946554f8ca99d7051136c2440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd893946554f8ca99d7051136c2440" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44357 } } }
I20260809 01:57:09.554363 3783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:09.554785 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53dd893946554f8ca99d7051136c2440. Latest consensus state: current_term: 1 leader_uuid: "53dd893946554f8ca99d7051136c2440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dd893946554f8ca99d7051136c2440" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 44357 } } }
I20260809 01:57:09.554946 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440 [sys.catalog]: This master's current role is: LEADER
W20260809 01:57:09.558193 3814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:57:09.558295 3814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:57:09.558475 3803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:09.559422 3803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:09.564781 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:09.554138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54755 (local address 127.3.220.62:44357)
0809 01:57:09.554277 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:09.554281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:09.555457 (+ 1176us) server_negotiation.cc:407] Connection header received
0809 01:57:09.555644 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:09.555650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:09.555701 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:09.555781 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:09.556470 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:09.557239 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:09.558442 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:09.558778 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:09.558903 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:09.560648 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:09.560652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:09.560655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:09.560681 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:09.562558 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:09.563060 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:09.563064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:09.563066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:09.563159 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:09.563487 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:09.563490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:09.563492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:09.563720 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:09.563863 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:09.564489 (+ 626us) server_negotiation.cc:299] Negotiation successful
0809 01:57:09.564613 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:57:09.566113 3803 catalog_manager.cc:1347] Generated new cluster ID: c777f5ce9040499181badf55eb750e63
I20260809 01:57:09.566177 3803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:09.588263 3803 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:09.589248 3803 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:09.597178 3803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53dd893946554f8ca99d7051136c2440: Generated new TSK 0
I20260809 01:57:09.597955 3803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:57:09.607396 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 3670
2026-08-09T01:57:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1662 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:09 dist-test-slave-gkw7 krb5kdc[3821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:09 dist-test-slave-gkw7 krb5kdc[3821](info): set up 2 sockets
Aug 09 01:57:09 dist-test-slave-gkw7 krb5kdc[3821](info): commencing operation
krb5kdc: starting...
W20260809 01:57:10.839716 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:10 dist-test-slave-gkw7 krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:57:10.969062 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39595
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:11.083385 3840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:11.083665 3840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:11.083748 3840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:11.087836 3840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:11.087901 3840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:11.087921 3840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:11.087939 3840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:11.087954 3840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:11.093331 3840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:11.094569 3840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:11.095669 3840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 01:57:11.104928 3840 server_base.cc:1034] running on GCE node
W20260809 01:57:11.104897 3852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:11.104875 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:11.104875 3848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:11.105844 3840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:11.106945 3840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:11.108111 3840 hybrid_clock.cc:648] HybridClock initialized: now 1786240631108082 us; error 46 us; skew 500 ppm
Aug 09 01:57:11 dist-test-slave-gkw7 krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:11.115772 3840 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:57:11.116878 3840 webserver.cc:460] Webserver started at http://127.3.220.62:38339/ using document root <none> and password file <none>
I20260809 01:57:11.117501 3840 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:11.117655 3840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:11.117929 3840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:11.120131 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "5a5b083fbc714221af0086b1fe22d02e"
format_stamp: "Formatted at 2026-08-09 01:57:11 on dist-test-slave-gkw7"
server_key: "a561a27dfd276863cdaaf9779025dc37"
server_key_iv: "0817cc9bd490e587c5a7a908eecb4924"
server_key_version: "encryptionkey@0"
I20260809 01:57:11.120751 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "5a5b083fbc714221af0086b1fe22d02e"
format_stamp: "Formatted at 2026-08-09 01:57:11 on dist-test-slave-gkw7"
server_key: "a561a27dfd276863cdaaf9779025dc37"
server_key_iv: "0817cc9bd490e587c5a7a908eecb4924"
server_key_version: "encryptionkey@0"
I20260809 01:57:11.124771 3840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 01:57:11.127542 3859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:11.128844 3840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:57:11.129002 3840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "5a5b083fbc714221af0086b1fe22d02e"
format_stamp: "Formatted at 2026-08-09 01:57:11 on dist-test-slave-gkw7"
server_key: "a561a27dfd276863cdaaf9779025dc37"
server_key_iv: "0817cc9bd490e587c5a7a908eecb4924"
server_key_version: "encryptionkey@0"
I20260809 01:57:11.129122 3840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:11.146944 3840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:11.150538 3840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:11.150770 3840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:11.158185 3840 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:39595
I20260809 01:57:11.158212 3935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:39595 every 8 connection(s)
I20260809 01:57:11.159220 3840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:11.162168 3937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:11.166260 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 3840
I20260809 01:57:11.166420 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:11.166716 3952 external_mini_cluster.cc:1428] Setting key 8f4b8857d70d4249e780d35dba0ff61d
I20260809 01:57:11.168272 3937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e: Bootstrap starting.
I20260809 01:57:11.170699 3937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:11.171770 3937 log.cc:824] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:11.174075 3937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e: No bootstrap required, opened a new log
I20260809 01:57:11.176707 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:11.168194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57483 (local address 127.3.220.62:39595)
0809 01:57:11.168661 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:11.168677 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:11.168715 (+ 38us) server_negotiation.cc:407] Connection header received
0809 01:57:11.169360 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:11.169378 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:11.169795 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:11.170138 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:11.170594 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:11.172221 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:11.173462 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:11.173988 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:11.174122 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:11.174985 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:11.174991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:11.175004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:11.175405 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:11.175461 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:11.175484 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:11.175561 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:11.175733 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 01:57:11.175968 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":148,"threads_started":1}
I20260809 01:57:11.178129 3937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b083fbc714221af0086b1fe22d02e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39595 } }
I20260809 01:57:11.178406 3937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:57:11.178479 3937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5b083fbc714221af0086b1fe22d02e, State: Initialized, Role: FOLLOWER
I20260809 01:57:11.178949 3937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b083fbc714221af0086b1fe22d02e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39595 } }
I20260809 01:57:11.179103 3937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:57:11.179176 3937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:57:11.179303 3937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:11.179750 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:11.180302 3937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b083fbc714221af0086b1fe22d02e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39595 } }
I20260809 01:57:11.180681 3937 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5b083fbc714221af0086b1fe22d02e; no voters:
I20260809 01:57:11.181044 3937 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:57:11.181144 3949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:11.181373 3949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [term 1 LEADER]: Becoming Leader. State: Replica: 5a5b083fbc714221af0086b1fe22d02e, State: Running, Role: LEADER
I20260809 01:57:11.181999 3949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b083fbc714221af0086b1fe22d02e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39595 } }
I20260809 01:57:11.182432 3937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:11.183588 3962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5b083fbc714221af0086b1fe22d02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b083fbc714221af0086b1fe22d02e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39595 } } }
I20260809 01:57:11.183732 3962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:11.184115 3963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5b083fbc714221af0086b1fe22d02e. Latest consensus state: current_term: 1 leader_uuid: "5a5b083fbc714221af0086b1fe22d02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5b083fbc714221af0086b1fe22d02e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 39595 } } }
I20260809 01:57:11.184238 3963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e [sys.catalog]: This master's current role is: LEADER
W20260809 01:57:11.187855 3984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:57:11.187938 3984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:57:11.188007 3978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:11.188880 3978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:11.189391 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:11.181871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39579 (local address 127.3.220.62:39595)
0809 01:57:11.181991 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:11.181995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:11.185074 (+ 3079us) server_negotiation.cc:407] Connection header received
0809 01:57:11.185187 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:11.185190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:11.185237 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:11.185315 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:11.185574 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:11.186320 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:11.187467 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:11.187789 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:11.187904 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:11.188898 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:11.188901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:11.188904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:11.188916 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:11.188948 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:11.188960 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:11.189047 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:11.189199 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 01:57:11.189261 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:57:11.194573 3978 catalog_manager.cc:1347] Generated new cluster ID: 732e657806b443a7a528ebad127102fb
I20260809 01:57:11.194658 3978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:11.203405 3978 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:11.204298 3978 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:11.214669 3978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5b083fbc714221af0086b1fe22d02e: Generated new TSK 0
I20260809 01:57:11.215417 3978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:57:11.222798 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 3840
2026-08-09T01:57:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:11 dist-test-slave-gkw7 krb5kdc[3991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:11 dist-test-slave-gkw7 krb5kdc[3991](info): set up 2 sockets
Aug 09 01:57:11 dist-test-slave-gkw7 krb5kdc[3991](info): commencing operation
krb5kdc: starting...
W20260809 01:57:12.458559 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:12 dist-test-slave-gkw7 krb5kdc[3991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:57:12.584105 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:42135
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:39223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:42135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:12.698446 4012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:12.698782 4012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:12.698868 4012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:12.703016 4012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:12.703115 4012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:12.703171 4012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:12.703230 4012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:12.703269 4012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:12.708669 4012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:39223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:42135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:42135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:12.709880 4012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:12.711001 4012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:12.719641 4021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:12.719628 4018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:12.719631 4019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:12.719870 4012 server_base.cc:1034] running on GCE node
I20260809 01:57:12.720520 4012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:12.721680 4012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:12.722849 4012 hybrid_clock.cc:648] HybridClock initialized: now 1786240632722829 us; error 32 us; skew 500 ppm
Aug 09 01:57:12 dist-test-slave-gkw7 krb5kdc[3991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:12.725716 4012 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:57:12.726642 4012 webserver.cc:460] Webserver started at http://127.3.220.62:36023/ using document root <none> and password file <none>
I20260809 01:57:12.727192 4012 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:12.727252 4012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:12.727428 4012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:12.729507 4012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "a0e9efe7d4d9427798ae8503b28927a2"
format_stamp: "Formatted at 2026-08-09 01:57:12 on dist-test-slave-gkw7"
server_key: "a79bb6ab364a93435cbc4b4484a5c7fb"
server_key_iv: "457057bc67a9139abc868c27b1380e8a"
server_key_version: "encryptionkey@0"
I20260809 01:57:12.730244 4012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "a0e9efe7d4d9427798ae8503b28927a2"
format_stamp: "Formatted at 2026-08-09 01:57:12 on dist-test-slave-gkw7"
server_key: "a79bb6ab364a93435cbc4b4484a5c7fb"
server_key_iv: "457057bc67a9139abc868c27b1380e8a"
server_key_version: "encryptionkey@0"
I20260809 01:57:12.734138 4012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 01:57:12.736559 4029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:12.737875 4012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:57:12.737979 4012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "a0e9efe7d4d9427798ae8503b28927a2"
format_stamp: "Formatted at 2026-08-09 01:57:12 on dist-test-slave-gkw7"
server_key: "a79bb6ab364a93435cbc4b4484a5c7fb"
server_key_iv: "457057bc67a9139abc868c27b1380e8a"
server_key_version: "encryptionkey@0"
I20260809 01:57:12.738066 4012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:12.742699 4012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:12.745481 4012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:12.745708 4012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:12.753089 4012 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:42135
I20260809 01:57:12.753265 4106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:42135 every 8 connection(s)
I20260809 01:57:12.754132 4012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:12.757000 4108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:12.761179 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4012
I20260809 01:57:12.761343 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:12.761685 3952 external_mini_cluster.cc:1428] Setting key 8db19c811c60b9697696616eae8fedd1
I20260809 01:57:12.763118 4108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2: Bootstrap starting.
I20260809 01:57:12.765923 4108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:12.766839 4108 log.cc:824] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:12.767365 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:12.763061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48805 (local address 127.3.220.62:42135)
0809 01:57:12.763484 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:12.763511 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:12.763553 (+ 42us) server_negotiation.cc:407] Connection header received
0809 01:57:12.764425 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:12.764452 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:12.764815 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:12.765182 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:12.765959 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:12.765966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:12.765984 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:12.765997 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:12.766064 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:12.766094 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:12.766233 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:12.766310 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 01:57:12.766461 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260809 01:57:12.768836 4108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2: No bootstrap required, opened a new log
I20260809 01:57:12.770083 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:12.773094 4108 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e9efe7d4d9427798ae8503b28927a2" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42135 } }
I20260809 01:57:12.773389 4108 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 01:57:12.773460 4108 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e9efe7d4d9427798ae8503b28927a2, State: Initialized, Role: FOLLOWER
I20260809 01:57:12.773940 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:12.772209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45131 (local address 127.3.220.62:42135)
0809 01:57:12.772324 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:12.772327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:12.772538 (+ 211us) server_negotiation.cc:407] Connection header received
0809 01:57:12.772660 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:12.772662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:12.772689 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:12.772754 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:12.773456 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:12.773461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:12.773463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 01:57:12.773477 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:12.773511 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 01:57:12.773562 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:12.773653 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:12.773744 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 01:57:12.773786 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 01:57:12.773969 4108 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e9efe7d4d9427798ae8503b28927a2" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42135 } }
I20260809 01:57:12.774160 4108 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 01:57:12.774228 4108 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 01:57:12.774379 4108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:12.775465 4108 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e9efe7d4d9427798ae8503b28927a2" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42135 } }
I20260809 01:57:12.775772 4108 leader_election.cc:302] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0e9efe7d4d9427798ae8503b28927a2; no voters:
I20260809 01:57:12.776017 4108 leader_election.cc:288] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 01:57:12.776161 4123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:12.776399 4123 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [term 1 LEADER]: Becoming Leader. State: Replica: a0e9efe7d4d9427798ae8503b28927a2, State: Running, Role: LEADER
I20260809 01:57:12.776746 4123 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e9efe7d4d9427798ae8503b28927a2" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42135 } }
I20260809 01:57:12.777248 4108 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:12.778432 4124 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0e9efe7d4d9427798ae8503b28927a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e9efe7d4d9427798ae8503b28927a2" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42135 } } }
I20260809 01:57:12.778604 4124 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:12.778950 4125 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0e9efe7d4d9427798ae8503b28927a2. Latest consensus state: current_term: 1 leader_uuid: "a0e9efe7d4d9427798ae8503b28927a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e9efe7d4d9427798ae8503b28927a2" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42135 } } }
I20260809 01:57:12.779035 4133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:12.779073 4125 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:12.782321 4133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:12.788192 4133 catalog_manager.cc:1347] Generated new cluster ID: ae646b4ad162439ab24947c588877bb5
I20260809 01:57:12.788280 4133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:12.817760 4133 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:12.818161 4133 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:12.823015 4133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0e9efe7d4d9427798ae8503b28927a2: Generated new TSK 0
I20260809 01:57:12.823845 4133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 01:57:12.829994 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4012
2026-08-09T01:57:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1608 ms)
[----------] 18 tests from AuthTokenIssuingTest (29286 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:12 dist-test-slave-gkw7 krb5kdc[4154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:12 dist-test-slave-gkw7 krb5kdc[4154](info): set up 2 sockets
Aug 09 01:57:12 dist-test-slave-gkw7 krb5kdc[4154](info): commencing operation
krb5kdc: starting...
W20260809 01:57:14.076759 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:57:14.207208 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:42623
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:42623,127.3.220.61:37979,127.3.220.60:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:14.337761 4172 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 01:57:14.338119 4172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:14.338196 4172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:14.338279 4172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:14.342412 4172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:14.342501 4172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:14.342540 4172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:14.342598 4172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:14.342638 4172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:14.348031 4172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:42623,127.3.220.61:37979,127.3.220.60:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:42623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:14.349156 4172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:14.350279 4172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:14.359184 4184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:14.359171 4182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:14.359292 4181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:14.359901 4172 server_base.cc:1034] running on GCE node
I20260809 01:57:14.360476 4172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:14.361672 4172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:14.362864 4172 hybrid_clock.cc:648] HybridClock initialized: now 1786240634362849 us; error 28 us; skew 500 ppm
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:14.365835 4172 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:57:14.366801 4172 webserver.cc:460] Webserver started at http://127.3.220.62:41607/ using document root <none> and password file <none>
I20260809 01:57:14.367334 4172 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:14.367393 4172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:14.367560 4172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:14.369663 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "b36c8baf04764a87a26e8804512ea8a5"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "b907bf3750c081d500b422f16baf4523"
server_key_iv: "fc93872ad0a043a91c569768f54b29ed"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.370436 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "b36c8baf04764a87a26e8804512ea8a5"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "b907bf3750c081d500b422f16baf4523"
server_key_iv: "fc93872ad0a043a91c569768f54b29ed"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.374449 4172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 01:57:14.376834 4194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:14.378208 4172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:57:14.378358 4172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "b36c8baf04764a87a26e8804512ea8a5"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "b907bf3750c081d500b422f16baf4523"
server_key_iv: "fc93872ad0a043a91c569768f54b29ed"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.378479 4172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:14.390257 4172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:14.393784 4172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:14.394022 4172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:14.402192 4172 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:42623
I20260809 01:57:14.402546 4272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:42623 every 8 connection(s)
I20260809 01:57:14.403261 4172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:14.404363 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4172
I20260809 01:57:14.404525 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:14.404830 3952 external_mini_cluster.cc:1428] Setting key 932d951d7aeaabff2a9e08db41856f09
I20260809 01:57:14.406499 4273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:14.412122 4273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.220.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.61@KRBTEST.COM" created.
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:57:14.430717 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.415146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42623 (local address 127.0.0.1:37742)
0809 01:57:14.415677 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.415730 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.416696 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.417491 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.417496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.418132 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.418499 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.418515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.420266 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.420272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.421548 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.421564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.422521 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.422524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.422693 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.423393 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.423421 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.424994 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.427441 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.427450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.427459 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.427691 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.428072 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.428074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.428075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.428187 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.428890 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.428896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.429024 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.429220 (+ 196us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.429637 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":150,"thread_start_us":156,"threads_started":1}
I20260809 01:57:14.430717 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.415114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.3.220.62:42623)
0809 01:57:14.415607 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.415622 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.415992 (+ 370us) server_negotiation.cc:407] Connection header received
0809 01:57:14.417000 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.417005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.417243 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.417426 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.418672 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.420107 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.421682 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.422336 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.422530 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.425172 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.425184 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.425204 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.425239 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.427318 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.427814 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.427824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.427827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.427971 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.428291 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.428294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.428296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.428776 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.428890 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.429284 (+ 394us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.429590 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":337,"spinlock_wait_cycles":6784,"thread_start_us":175,"threads_started":1}
I20260809 01:57:14.433611 4273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:14.434520 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.434236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36720)
0809 01:57:14.434384 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.434457 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 01:57:14.434746 4202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
I20260809 01:57:14.435714 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.435499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36732)
0809 01:57:14.435609 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.435662 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 01:57:14.435906 4273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37979: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab.
Entry for principal kudu/127.3.220.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab.
I20260809 01:57:14.437637 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.61:37979
--webserver_interface=127.3.220.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab
--principal=kudu/127.3.220.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:42623,127.3.220.61:37979,127.3.220.60:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:57:14.485164 4273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } attempt: 1
I20260809 01:57:14.486362 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.485906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36744)
0809 01:57:14.486130 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.486229 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 01:57:14.487262 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.486987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36750)
0809 01:57:14.487113 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.487178 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 01:57:14.487454 4273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37979: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
I20260809 01:57:14.555650 4273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } attempt: 2
I20260809 01:57:14.556502 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.556212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36754)
0809 01:57:14.556364 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.556427 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 01:57:14.557317 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.557089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36764)
0809 01:57:14.557203 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.557257 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 01:57:14.557461 4273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37979: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37979: connect: Connection refused (error 111)
W20260809 01:57:14.559873 4283 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 01:57:14.560205 4283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:14.560276 4283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:14.560348 4283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:14.564493 4283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:14.564584 4283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:14.564620 4283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:14.564658 4283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:14.564693 4283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:14.570149 4283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:42623,127.3.220.61:37979,127.3.220.60:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab
--principal=kudu/127.3.220.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.61:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.220.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:14.571305 4283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:14.572381 4283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:14.581302 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:14.581396 4283 server_base.cc:1034] running on GCE node
W20260809 01:57:14.581385 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:14.581298 4296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:14.582201 4283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:14.583454 4283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:14.584643 4283 hybrid_clock.cc:648] HybridClock initialized: now 1786240634584629 us; error 27 us; skew 500 ppm
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:14.587405 4283 init.cc:377] Logged in from keytab as kudu/127.3.220.61@KRBTEST.COM (short username kudu)
I20260809 01:57:14.588263 4283 webserver.cc:460] Webserver started at http://127.3.220.61:34319/ using document root <none> and password file <none>
I20260809 01:57:14.588841 4283 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:14.588932 4283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:14.589169 4283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:14.591280 4283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/data/instance:
uuid: "d3530888239e40a1bdab7eef40e176b9"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "31c2d2b3f4ff5447ae120c8f9430095a"
server_key_iv: "3409db814e0ddd396d3513eaec989a8c"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.591868 4283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/wal/instance:
uuid: "d3530888239e40a1bdab7eef40e176b9"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "31c2d2b3f4ff5447ae120c8f9430095a"
server_key_iv: "3409db814e0ddd396d3513eaec989a8c"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.595683 4283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 01:57:14.598181 4307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:14.599888 4283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:57:14.600031 4283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/wal
uuid: "d3530888239e40a1bdab7eef40e176b9"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "31c2d2b3f4ff5447ae120c8f9430095a"
server_key_iv: "3409db814e0ddd396d3513eaec989a8c"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.600155 4283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:14.625705 4283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:14.628836 4283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:14.629076 4283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:14.636240 4283 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.61:37979
I20260809 01:57:14.636540 4379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.61:37979 every 8 connection(s)
I20260809 01:57:14.637284 4283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
I20260809 01:57:14.640121 4380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:14.644039 4380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:14.646109 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4283
I20260809 01:57:14.646265 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-1/wal/instance
I20260809 01:57:14.646551 3952 external_mini_cluster.cc:1428] Setting key 1be8f899ded57e6d843826a5be1a2370
I20260809 01:57:14.648653 4273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } attempt: 3
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.60@KRBTEST.COM" created.
I20260809 01:57:14.660502 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.649128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36778)
0809 01:57:14.649359 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.649376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.649461 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.650100 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.650103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.650303 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.650401 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.650406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.651501 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.651508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.652100 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.652112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.652898 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.652903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.653097 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.653710 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.653732 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.655628 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.658348 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.658351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.658352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.658667 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.659012 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.659017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.659019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.659136 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.659924 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.659927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.660047 (+ 120us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.660168 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.660325 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":25}
I20260809 01:57:14.661631 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.645992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.3.220.62:42623)
0809 01:57:14.646165 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.646170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.647151 (+ 981us) server_negotiation.cc:407] Connection header received
0809 01:57:14.648124 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.648131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.648204 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.648314 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.650118 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.650970 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.652655 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.653031 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.653138 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.656359 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.656362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.656364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.656385 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.658553 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.659125 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.659128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.659130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.659205 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.659530 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.659533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.659534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.659795 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.659916 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.661213 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.661360 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":896}
I20260809 01:57:14.661754 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.646452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42623 (local address 127.0.0.1:37758)
0809 01:57:14.646963 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.647018 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.647914 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.648534 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.648546 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.649098 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.649923 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.649947 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.651127 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.651133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.652526 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.652539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.653182 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.653186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.653300 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.653900 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.653916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.656203 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.658658 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.658669 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.658680 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.659033 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.659299 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.659304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.659306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.659439 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.660839 (+ 1400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.660845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.660958 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.661109 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.661277 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":1,"thread_start_us":171,"threads_started":1}
I20260809 01:57:14.661754 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.649194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.3.220.61:37979)
0809 01:57:14.649728 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.649737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.649756 (+ 19us) server_negotiation.cc:407] Connection header received
0809 01:57:14.649792 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.649796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.649943 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.650107 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.650536 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.651382 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.652231 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.652757 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.652914 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.655782 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.655789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.655802 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.655828 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.658226 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.658789 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.658794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.658795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.658908 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.659231 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.659234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.659235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.659811 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.659958 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.660369 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.660671 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":450,"thread_start_us":146,"threads_started":1}
I20260809 01:57:14.663436 4380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:14.663793 4273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:14.664480 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.664244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39838)
0809 01:57:14.664349 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.664412 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 01:57:14.665316 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.665011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39844)
0809 01:57:14.665186 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.665245 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260809 01:57:14.665486 4273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:39873: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:57:14.672309 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.664155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36790)
0809 01:57:14.664247 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.664263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.664337 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.664797 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.664801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.664989 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.665117 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.665124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.665998 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.666004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.666531 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.666539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.666998 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.667001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.667089 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.667620 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.667631 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.669099 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.670849 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.670852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.670854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.671122 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.671445 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.671448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.671449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.671534 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.671887 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.671889 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.671964 (+ 75us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.672065 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.672180 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 01:57:14.672348 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.664429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.3.220.61:37979)
0809 01:57:14.664534 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.664540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.664551 (+ 11us) server_negotiation.cc:407] Connection header received
0809 01:57:14.664584 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.664587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.664627 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.664700 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.665223 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.665892 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.666644 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.666890 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.666999 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.669257 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.669264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.669270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.669300 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.670743 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.671226 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.671231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.671234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.671327 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.671615 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.671617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.671618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.671809 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.671890 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.672099 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.672236 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":32}
I20260809 01:57:14.673168 4380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:14.673951 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.673679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39850)
0809 01:57:14.673811 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.673901 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 01:57:14.674593 4308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
I20260809 01:57:14.675524 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.675339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39860)
0809 01:57:14.675436 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.675479 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 01:57:14.675684 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:39873: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab.
Entry for principal kudu/127.3.220.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab.
I20260809 01:57:14.676800 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.60:39873
--webserver_interface=127.3.220.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab
--principal=kudu/127.3.220.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:42623,127.3.220.61:37979,127.3.220.60:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:57:14.696691 4273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } attempt: 1
I20260809 01:57:14.697636 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.697274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39870)
0809 01:57:14.697434 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.697498 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 01:57:14.698369 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.698164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39886)
0809 01:57:14.698263 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.698314 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 01:57:14.698513 4273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:39873: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
I20260809 01:57:14.725008 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } attempt: 1
I20260809 01:57:14.726017 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.725662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39888)
0809 01:57:14.725841 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.725919 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 01:57:14.726857 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.726615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39892)
0809 01:57:14.726730 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.726784 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 01:57:14.727011 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:39873: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
I20260809 01:57:14.773682 4273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } attempt: 2
I20260809 01:57:14.774602 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.774272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39894)
0809 01:57:14.774456 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.774521 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 01:57:14.775404 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.775194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39910)
0809 01:57:14.775299 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.775346 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 01:57:14.775564 4273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:39873: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
I20260809 01:57:14.795176 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } attempt: 2
I20260809 01:57:14.796016 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.795699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39916)
0809 01:57:14.795864 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.795929 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 01:57:14.796761 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.796555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39918)
0809 01:57:14.796658 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.796707 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 01:57:14.796914 4380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:39873: Network error: Client connection negotiation failed: client connection to 127.3.220.60:39873: connect: Connection refused (error 111)
W20260809 01:57:14.797124 4392 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 01:57:14.797417 4392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:14.797473 4392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:14.797561 4392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:14.801457 4392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:14.801586 4392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:14.801632 4392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:14.801690 4392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:14.801728 4392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:14.808179 4392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:42623,127.3.220.61:37979,127.3.220.60:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab
--principal=kudu/127.3.220.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.60:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.220.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:14.809289 4392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:14.810415 4392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:14.819110 4401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:14.819224 4398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:14.819224 4399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:14.819401 4392 server_base.cc:1034] running on GCE node
I20260809 01:57:14.820020 4392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:14.821110 4392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:14.822299 4392 hybrid_clock.cc:648] HybridClock initialized: now 1786240634822279 us; error 32 us; skew 500 ppm
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:14.825183 4392 init.cc:377] Logged in from keytab as kudu/127.3.220.60@KRBTEST.COM (short username kudu)
I20260809 01:57:14.826128 4392 webserver.cc:460] Webserver started at http://127.3.220.60:35101/ using document root <none> and password file <none>
I20260809 01:57:14.826701 4392 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:14.826797 4392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:14.827049 4392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:14.829115 4392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/data/instance:
uuid: "ccf0c8849fd045cfb5af0bbac2060cd5"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "fa8a70302e48d676285911fff668e940"
server_key_iv: "2d751dcbcc0571c2b8af8072f2ed6b37"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.829768 4392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/wal/instance:
uuid: "ccf0c8849fd045cfb5af0bbac2060cd5"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "fa8a70302e48d676285911fff668e940"
server_key_iv: "2d751dcbcc0571c2b8af8072f2ed6b37"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.833658 4392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 01:57:14.836356 4416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:14.837679 4392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:57:14.837815 4392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/wal
uuid: "ccf0c8849fd045cfb5af0bbac2060cd5"
format_stamp: "Formatted at 2026-08-09 01:57:14 on dist-test-slave-gkw7"
server_key: "fa8a70302e48d676285911fff668e940"
server_key_iv: "2d751dcbcc0571c2b8af8072f2ed6b37"
server_key_version: "encryptionkey@0"
I20260809 01:57:14.837939 4392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:14.860651 4392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:14.863935 4392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:14.864156 4392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:14.871387 4392 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.60:39873
I20260809 01:57:14.871721 4501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.60:39873 every 8 connection(s)
I20260809 01:57:14.872391 4392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
I20260809 01:57:14.874747 4273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } attempt: 3
I20260809 01:57:14.875452 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4392
I20260809 01:57:14.875583 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786240338176169-3952-0/minicluster-data/master-2/wal/instance
I20260809 01:57:14.875700 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:14.875892 3952 external_mini_cluster.cc:1428] Setting key d0a05a1a0462fc5c02733bd5dc42c36a
I20260809 01:57:14.880133 4502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
I20260809 01:57:14.888084 4380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } attempt: 3
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:57:14.890697 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.875312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39930)
0809 01:57:14.875434 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.875452 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.875552 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.878380 (+ 2828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.878384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.878606 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.878742 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.878751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.881333 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.881337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.882225 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.882237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.883297 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.883301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.883382 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.883901 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.883912 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.885432 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.888960 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.888966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.888972 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.889245 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.889585 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.889588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.889590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.889701 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.890157 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.890160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.890265 (+ 105us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.890390 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.890533 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 01:57:14.891681 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.876099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.3.220.60:39873)
0809 01:57:14.876803 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.876822 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.876866 (+ 44us) server_negotiation.cc:407] Connection header received
0809 01:57:14.877579 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.877607 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.878044 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.878417 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.879476 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.881184 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.882380 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.883176 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.883352 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.885650 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.885657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.885670 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.885704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.888833 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.889365 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.889368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.889369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.889440 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.889811 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.889815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.889816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.890033 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.890156 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.890547 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.890861 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1223,"server-negotiator.queue_time_us":542,"spinlock_wait_cycles":19712,"thread_start_us":160,"threads_started":1}
I20260809 01:57:14.893049 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.876664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.3.220.60:39873)
0809 01:57:14.877725 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.877734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.877747 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:57:14.877783 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.877786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.878044 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.878417 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.879413 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.881184 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.882848 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.883193 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.883349 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.886146 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.886150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.886152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.886182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.888207 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.888763 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.888773 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.888780 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.888933 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.889252 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.889255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.889257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.889873 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.890075 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.892707 (+ 2632us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.892821 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":259,"server-negotiator.queue_time_us":999,"spinlock_wait_cycles":12416,"thread_start_us":75,"threads_started":1}
I20260809 01:57:14.893863 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.881889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.3.220.62:42623)
0809 01:57:14.881997 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.882001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.882739 (+ 738us) server_negotiation.cc:407] Connection header received
0809 01:57:14.882851 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.882854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.882902 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.882979 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.884397 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.885150 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.886297 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.886653 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.886778 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.889521 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.889566 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.889571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.889597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.891300 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.892167 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.892171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.892172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.892265 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.892575 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.892579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.892580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.892882 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.892979 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.893602 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.893721 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:57:14.893877 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.881858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:42623 (local address 127.0.0.1:37772)
0809 01:57:14.882623 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.882660 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.882764 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.883055 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.883060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.883544 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.884237 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.884250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.885271 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.885274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.886186 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.886199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.886776 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.886779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.886869 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.887419 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.887439 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.889378 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.891636 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.891645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.891650 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.891955 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.892368 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.892370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.892372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.892465 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.893237 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.893244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.893356 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.893506 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.893713 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":122,"threads_started":1}
I20260809 01:57:14.895265 4502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
I20260809 01:57:14.897316 4273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5: Bootstrap starting.
I20260809 01:57:14.900038 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.888562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39956)
0809 01:57:14.888737 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.888751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.888832 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.890775 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.890782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.890973 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.891102 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.891110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.892271 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.892275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.893031 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.893043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.893816 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.893829 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.893918 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.894687 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.894700 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.896131 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.898407 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.898411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.898413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.898663 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.899022 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.899026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.899028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.899121 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.899540 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.899544 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.899639 (+ 95us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.899730 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.899872 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 01:57:14.900415 4273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:14.900717 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.889032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.3.220.60:39873)
0809 01:57:14.890251 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.890260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.890274 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:57:14.890311 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.890314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.890368 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.891241 (+ 873us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.891354 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.892110 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.893293 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.893654 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.894137 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.896595 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.896598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.896601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.896622 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.898279 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.898803 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.898806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.898808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.898911 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.899228 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.899231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.899233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.899425 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.900310 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.900408 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.900524 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1125,"thread_start_us":102,"threads_started":1}
I20260809 01:57:14.901572 4273 log.cc:824] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:57:14.903919 4273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5: No bootstrap required, opened a new log
I20260809 01:57:14.904067 4380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9: Bootstrap starting.
I20260809 01:57:14.906389 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.895720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37979 (local address 127.0.0.1:36802)
0809 01:57:14.896134 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.896151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.896235 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.896513 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.896518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.896698 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.896806 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.896812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.897945 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.897948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.898782 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.898793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.899480 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.899484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.899578 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.900320 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.900351 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.902654 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.904642 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.904645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.904647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.904894 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.905226 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.905230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.905233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.905338 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.905889 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.905892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.905986 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.906092 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.906243 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339}
I20260809 01:57:14.906850 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.895951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.3.220.61:37979)
0809 01:57:14.896306 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.896310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.896320 (+ 10us) server_negotiation.cc:407] Connection header received
0809 01:57:14.896352 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.896355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.896395 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.896472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.896945 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.897811 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.898929 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.899271 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.899367 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.902818 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.902830 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.902833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.902859 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.904523 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.905023 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.905026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.905028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.905116 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.905444 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.905448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.905449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.905732 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.905839 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.906582 (+ 743us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.906693 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260809 01:57:14.906924 4380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:14.907672 4502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:14.907810 4380 log.cc:824] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:14.908197 4273 raft_consensus.cc:348] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } }
I20260809 01:57:14.908442 4273 raft_consensus.cc:729] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36c8baf04764a87a26e8804512ea8a5, State: Initialized, Role: FOLLOWER
I20260809 01:57:14.908793 4273 consensus_queue.cc:260] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } }
I20260809 01:57:14.910032 4380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9: No bootstrap required, opened a new log
I20260809 01:57:14.910238 4273 sys_catalog.cc:564] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:14.910753 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.895366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.3.220.62:42623)
0809 01:57:14.895466 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.895470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.895482 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:57:14.895514 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.895517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.895561 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.895655 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.896608 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.897858 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.898957 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.899296 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.899367 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.906144 (+ 6777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.906148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.906150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.906176 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.908705 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.909217 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.909220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.909222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.909311 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.909735 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.909738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.909740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.909952 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.910103 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.910490 (+ 387us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.910604 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
I20260809 01:57:14.915372 4380 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } }
I20260809 01:57:14.915629 4380 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3530888239e40a1bdab7eef40e176b9, State: Initialized, Role: FOLLOWER
I20260809 01:57:14.915954 4380 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } }
I20260809 01:57:14.917320 4380 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:14.918232 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.908336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.3.220.60:39873)
0809 01:57:14.908434 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.908438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.908668 (+ 230us) server_negotiation.cc:407] Connection header received
0809 01:57:14.908799 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.908803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.908858 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.908907 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.909493 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.910269 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.911611 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.912009 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.912110 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.914563 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.914567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.914569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.914593 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.916361 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.916909 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.916912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.916914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.916981 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.917239 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.917242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.917243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.917427 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.917493 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.917912 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.918039 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 01:57:14.918941 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.908148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:39873 (local address 127.0.0.1:39962)
0809 01:57:14.908607 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:14.908621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:14.908718 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:14.909139 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:14.909141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:14.909300 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:14.909383 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.909389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.910452 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.910455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.911486 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:14.911498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.912246 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.912250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:14.912351 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.912840 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:14.912855 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:14.914425 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:14.916558 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.916560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.916562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.916803 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.917085 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:14.917088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:14.917090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:14.917159 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:14.917657 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:14.917661 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:14.917742 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 01:57:14.918652 (+ 910us) client_negotiation.cc:240] Negotiation successful
0809 01:57:14.918783 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400}
I20260809 01:57:14.920190 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.920331 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: This master's current role is: FOLLOWER
Aug 09 01:57:14 dist-test-slave-gkw7 krb5kdc[4154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:57:14.922222 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5: Bootstrap starting.
W20260809 01:57:14.923900 4535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:57:14.923990 4535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:57:14.925050 4502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:14.925999 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.914806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.3.220.61:37979)
0809 01:57:14.914912 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.914916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.914928 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:57:14.915025 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.915028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.915068 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.915909 (+ 841us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.915999 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.916702 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.917827 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.918188 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.921201 (+ 3013us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.921313 (+ 112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.921316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.921318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.921337 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.923045 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.923563 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.923568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.923571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.923661 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.923997 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.924000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.924002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.924208 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.925631 (+ 1423us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.925718 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.925852 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 01:57:14.915429 4172 master_runner.cc:186] Error getting master registration for 127.3.220.61:37979: OK, instance_id { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" instance_seqno: 1786240634634169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 01:57:14.927793 4172 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:57:14.929322 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:14.929289 4502 log.cc:824] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5: Log is configured to *not* fsync() on all Append() calls
W20260809 01:57:14.922845 4559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:57:14.930532 4559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:57:14.924201 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.930713 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:14.930905 4237 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b36c8baf04764a87a26e8804512ea8a5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37786
I20260809 01:57:14.931205 4237 raft_consensus.cc:482] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 01:57:14.931329 4237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:14.932710 4237 raft_consensus.cc:504] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } }
I20260809 01:57:14.930522 4283 master_runner.cc:186] Error getting master registration for 127.3.220.60:39873: OK, instance_id { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" instance_seqno: 1786240634869315 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 01:57:14.933250 4283 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:57:14.933781 4237 leader_election.cc:288] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [CANDIDATE]: Term 1 election: Requested vote from peers d3530888239e40a1bdab7eef40e176b9 (127.3.220.61:37979), ccf0c8849fd045cfb5af0bbac2060cd5 (127.3.220.60:39873)
I20260809 01:57:14.934279 4347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b36c8baf04764a87a26e8804512ea8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3530888239e40a1bdab7eef40e176b9"
I20260809 01:57:14.934525 4347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:14.936394 4347 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b36c8baf04764a87a26e8804512ea8a5 in term 1.
I20260809 01:57:14.936831 4202 leader_election.cc:302] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36c8baf04764a87a26e8804512ea8a5, d3530888239e40a1bdab7eef40e176b9; no voters:
I20260809 01:57:14.937108 4514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:14.937381 4514 raft_consensus.cc:686] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [term 1 LEADER]: Becoming Leader. State: Replica: b36c8baf04764a87a26e8804512ea8a5, State: Running, Role: LEADER
I20260809 01:57:14.937762 4514 consensus_queue.cc:237] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } }
I20260809 01:57:14.940083 4569 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b36c8baf04764a87a26e8804512ea8a5. Latest consensus state: current_term: 1 leader_uuid: "b36c8baf04764a87a26e8804512ea8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.940224 4569 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:14.940526 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5: No bootstrap required, opened a new log
I20260809 01:57:14.945792 4571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:14.946765 4571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 01:57:14.945489 4502 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } }
I20260809 01:57:14.948985 4502 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0c8849fd045cfb5af0bbac2060cd5, State: Initialized, Role: FOLLOWER
I20260809 01:57:14.949333 4502 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } }
I20260809 01:57:14.950698 4502 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:14.950776 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b36c8baf04764a87a26e8804512ea8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5"
I20260809 01:57:14.951020 4451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:14.952339 4451 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b36c8baf04764a87a26e8804512ea8a5 in term 1.
I20260809 01:57:14.953740 4573 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.953882 4573 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:14.955464 4347 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [term 1 FOLLOWER]: Refusing update from remote peer b36c8baf04764a87a26e8804512ea8a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:57:14.956382 4514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 01:57:14.958779 4590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:57:14.958853 4590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:57:14.959342 4451 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [term 1 FOLLOWER]: Refusing update from remote peer b36c8baf04764a87a26e8804512ea8a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:57:14.960253 4569 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:57:14.963235 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b36c8baf04764a87a26e8804512ea8a5. Latest consensus state: current_term: 1 leader_uuid: "b36c8baf04764a87a26e8804512ea8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.963364 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:14.963733 4573 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b36c8baf04764a87a26e8804512ea8a5. Latest consensus state: current_term: 1 leader_uuid: "b36c8baf04764a87a26e8804512ea8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.963846 4573 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:14.972344 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.953933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.3.220.62:42623)
0809 01:57:14.954041 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.954045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.963039 (+ 8994us) server_negotiation.cc:407] Connection header received
0809 01:57:14.963280 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.963283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.963334 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.963393 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.964449 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.965244 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.966421 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.966719 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.966795 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.968813 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.968817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.968830 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.968857 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.970496 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.971005 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.971009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.971011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.971094 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.971381 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.971385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.971387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.971589 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.971702 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.972063 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.972184 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 01:57:14.973932 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.954350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.3.220.60:39873)
0809 01:57:14.954802 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.954806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.964603 (+ 9797us) server_negotiation.cc:407] Connection header received
0809 01:57:14.964763 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.964766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.964808 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.964856 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.965830 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.966600 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.967737 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.968086 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.968163 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.970382 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.970386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.970389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.970416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.972072 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.972583 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.972586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.972588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.972671 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.972957 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.972961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.972962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.973157 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.973303 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.973654 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.973778 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378}
I20260809 01:57:14.975567 4569 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b36c8baf04764a87a26e8804512ea8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.975725 4569 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:14.977746 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b36c8baf04764a87a26e8804512ea8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.977872 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:14.978263 4596 mvcc.cc:205] Tried to move back new op lower bound from 7316441640768577536 to 7316441640708698112. Current Snapshot: MvccSnapshot[applied={T|T < 7316441640768577536}]
I20260809 01:57:14.982860 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:14.953389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.3.220.61:37979)
0809 01:57:14.953517 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:14.953522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:14.973450 (+ 19928us) server_negotiation.cc:407] Connection header received
0809 01:57:14.973686 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:14.973689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:14.973754 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:14.973811 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:14.974900 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.975592 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.976705 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:14.977022 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:14.977099 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:14.979190 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:14.979194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:14.979197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:14.979222 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:14.980891 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.981362 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.981366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.981367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.981455 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:14.981830 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:14.981834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:14.981836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:14.982078 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:14.982196 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:14.982564 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 01:57:14.982682 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 01:57:14.988026 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b36c8baf04764a87a26e8804512ea8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.988152 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5 [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:14.989432 4573 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b36c8baf04764a87a26e8804512ea8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36c8baf04764a87a26e8804512ea8a5" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 42623 } } peers { permanent_uuid: "d3530888239e40a1bdab7eef40e176b9" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37979 } } peers { permanent_uuid: "ccf0c8849fd045cfb5af0bbac2060cd5" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 39873 } } }
I20260809 01:57:14.989571 4573 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:14.991259 4571 catalog_manager.cc:1347] Generated new cluster ID: 5ba9eadfb58745d59144b7bfb5e60689
I20260809 01:57:14.991328 4571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:15.006696 4571 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:15.007984 4571 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:15.021709 4571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b36c8baf04764a87a26e8804512ea8a5: Generated new TSK 0
I20260809 01:57:15.022711 4571 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 01:57:15.059288 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.056970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.3.220.61:37979)
0809 01:57:15.057098 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.057103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.058172 (+ 1069us) server_negotiation.cc:407] Connection header received
0809 01:57:15.058378 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.058382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.058433 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.058487 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.059148 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36828: BlockingRecv error: recv got EOF from 127.0.0.1:36828 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 01:57:15.060979 4278 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.057876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.3.220.62:42623)
0809 01:57:15.057995 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.057999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.060059 (+ 2060us) server_negotiation.cc:407] Connection header received
0809 01:57:15.060230 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.060233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.060281 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.060334 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.060850 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37806: BlockingRecv error: recv got EOF from 127.0.0.1:37806 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 01:57:15.063148 4511 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.061792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.3.220.60:39873)
0809 01:57:15.061942 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.061946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.062258 (+ 312us) server_negotiation.cc:407] Connection header received
0809 01:57:15.062423 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.062426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.062474 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.062526 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.063023 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39970: BlockingRecv error: recv got EOF from 127.0.0.1:39970 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 01:57:15.067669 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.066153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.3.220.61:37979)
0809 01:57:15.066255 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.066259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.066734 (+ 475us) server_negotiation.cc:407] Connection header received
0809 01:57:15.066964 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.066967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.067039 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.067088 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.067568 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36836: BlockingRecv error: recv got EOF from 127.0.0.1:36836 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 01:57:15.072840 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.071288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.3.220.61:37979)
0809 01:57:15.071400 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.071404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.071945 (+ 541us) server_negotiation.cc:407] Connection header received
0809 01:57:15.072125 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.072128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.072181 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.072233 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.072724 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36842: BlockingRecv error: recv got EOF from 127.0.0.1:36842 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 01:57:15.078461 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.076889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.3.220.61:37979)
0809 01:57:15.076987 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.076991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.077520 (+ 529us) server_negotiation.cc:407] Connection header received
0809 01:57:15.077737 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.077740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.077789 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.077850 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.078366 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36850: BlockingRecv error: recv got EOF from 127.0.0.1:36850 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 01:57:15.086304 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.084598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.3.220.61:37979)
0809 01:57:15.084732 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.084737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.085313 (+ 576us) server_negotiation.cc:407] Connection header received
0809 01:57:15.085505 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.085511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.085588 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.085653 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.086202 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36862: BlockingRecv error: recv got EOF from 127.0.0.1:36862 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 01:57:15.098713 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.096686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.3.220.61:37979)
0809 01:57:15.096805 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.096809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.097474 (+ 665us) server_negotiation.cc:407] Connection header received
0809 01:57:15.097893 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.097897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.097961 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.098026 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.098592 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36866: BlockingRecv error: recv got EOF from 127.0.0.1:36866 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 01:57:15.119146 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.117362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.3.220.61:37979)
0809 01:57:15.117484 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.117488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.118132 (+ 644us) server_negotiation.cc:407] Connection header received
0809 01:57:15.118349 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.118353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.118407 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.118459 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.119026 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36872: BlockingRecv error: recv got EOF from 127.0.0.1:36872 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 01:57:15.155166 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.153738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.3.220.61:37979)
0809 01:57:15.153884 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.153890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.154183 (+ 293us) server_negotiation.cc:407] Connection header received
0809 01:57:15.154380 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.154383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.154431 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.154524 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.155041 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36878: BlockingRecv error: recv got EOF from 127.0.0.1:36878 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 01:57:15.222631 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.221220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36886 (local address 127.3.220.61:37979)
0809 01:57:15.221411 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.221416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.221638 (+ 222us) server_negotiation.cc:407] Connection header received
0809 01:57:15.221833 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.221845 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.221896 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.221991 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.222482 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36886: BlockingRecv error: recv got EOF from 127.0.0.1:36886 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 01:57:15.354215 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.353009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.3.220.61:37979)
0809 01:57:15.353171 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.353175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.353335 (+ 160us) server_negotiation.cc:407] Connection header received
0809 01:57:15.353480 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.353483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.353564 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.353664 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.354104 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36900: BlockingRecv error: recv got EOF from 127.0.0.1:36900 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 01:57:15.613852 4387 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:15.612595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.3.220.61:37979)
0809 01:57:15.612758 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:15.612765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:15.612868 (+ 103us) server_negotiation.cc:407] Connection header received
0809 01:57:15.613054 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:15.613057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:15.613113 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:15.613204 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:15.613713 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36908: BlockingRecv error: recv got EOF from 127.0.0.1:36908 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 01:57:15.931952 4559 catalog_manager.cc:1259] Loaded cluster ID: 5ba9eadfb58745d59144b7bfb5e60689
I20260809 01:57:15.932047 4559 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9: loading cluster ID for follower catalog manager: success
I20260809 01:57:15.933790 4559 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9: acquiring CA information for follower catalog manager: success
I20260809 01:57:15.934986 4559 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3530888239e40a1bdab7eef40e176b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:57:15.960196 4590 catalog_manager.cc:1259] Loaded cluster ID: 5ba9eadfb58745d59144b7bfb5e60689
I20260809 01:57:15.960310 4590 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5: loading cluster ID for follower catalog manager: success
I20260809 01:57:15.962064 4590 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5: acquiring CA information for follower catalog manager: success
I20260809 01:57:15.963248 4590 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccf0c8849fd045cfb5af0bbac2060cd5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:57:16.132350 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:16.128102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.3.220.61:37979)
0809 01:57:16.128402 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:16.128409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:16.128427 (+ 18us) server_negotiation.cc:407] Connection header received
0809 01:57:16.128613 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:16.128616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:16.128780 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:16.128915 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:16.129218 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.129812 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.130747 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.130987 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.131103 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:16.131470 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:16.131923 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:16.132069 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:16.132108 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 01:57:16.132182 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":72,"threads_started":1}
I20260809 01:57:16.139897 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:16.135893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.3.220.60:39873)
0809 01:57:16.136132 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:16.136137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:16.136154 (+ 17us) server_negotiation.cc:407] Connection header received
0809 01:57:16.136258 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:16.136261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:16.136376 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:16.136507 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:16.136734 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.137259 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.138348 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.138580 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.138693 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:16.139023 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:16.139462 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:16.139603 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:16.139670 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 01:57:16.139729 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
I20260809 01:57:16.147486 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:16.143127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.3.220.61:37979)
0809 01:57:16.143329 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:16.143336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:16.143832 (+ 496us) server_negotiation.cc:407] Connection header received
0809 01:57:16.143918 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:16.143921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:16.143971 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:16.144051 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:16.144391 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.145195 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.146224 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.146555 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.146667 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:16.146966 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:16.147059 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:16.147132 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:16.147276 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 01:57:16.147338 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260809 01:57:16.148236 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:16.143588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.3.220.60:39873)
0809 01:57:16.143713 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:16.143718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:16.144072 (+ 354us) server_negotiation.cc:407] Connection header received
0809 01:57:16.144206 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:16.144209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:16.144257 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:16.144357 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:16.144600 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.145100 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.146359 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.146692 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.146817 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:16.147076 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:16.147171 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:16.147246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:16.148027 (+ 781us) server_negotiation.cc:299] Negotiation successful
0809 01:57:16.148081 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 01:57:16.148352 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:16.143429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.3.220.62:42623)
0809 01:57:16.143668 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:16.143673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:16.143686 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:57:16.143785 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:16.143789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:16.143963 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:16.144343 (+ 380us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:16.144460 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.145233 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.146486 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:16.146805 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:16.146954 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:16.147503 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:16.147851 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:16.147950 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:16.148118 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 01:57:16.148182 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260809 01:57:16.155903 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4172
I20260809 01:57:16.167434 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4283
I20260809 01:57:16.179406 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4392
2026-08-09T01:57:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3354 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 01:57:16 dist-test-slave-gkw7 krb5kdc[4737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 01:57:16 dist-test-slave-gkw7 krb5kdc[4737](info): set up 2 sockets
Aug 09 01:57:16 dist-test-slave-gkw7 krb5kdc[4737](info): commencing operation
krb5kdc: starting...
W20260809 01:57:17.420049 3952 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/test-user.keytab.
Aug 09 01:57:17 dist-test-slave-gkw7 krb5kdc[4737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T01:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T01:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.220.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
Entry for principal kudu/127.3.220.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab.
I20260809 01:57:17.553360 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41169
--webserver_interface=127.3.220.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:41169,127.3.220.61:37269,127.3.220.60:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 01:57:17.667380 4756 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 01:57:17.667654 4756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:17.667678 4756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:17.667793 4756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:17.671783 4756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:17.671841 4756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:17.671861 4756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:17.671877 4756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:17.671893 4756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:17.677273 4756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:41169,127.3.220.61:37269,127.3.220.60:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.62.keytab
--principal=kudu/127.3.220.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.62:41169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.220.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:17.678454 4756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:17.679447 4756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:17.688103 4762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:17.688174 4766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:17.688081 4764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:17.688305 4756 server_base.cc:1034] running on GCE node
I20260809 01:57:17.688983 4756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:17.690140 4756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:17.691300 4756 hybrid_clock.cc:648] HybridClock initialized: now 1786240637691269 us; error 41 us; skew 500 ppm
Aug 09 01:57:17 dist-test-slave-gkw7 krb5kdc[4737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:17.694203 4756 init.cc:377] Logged in from keytab as kudu/127.3.220.62@KRBTEST.COM (short username kudu)
I20260809 01:57:17.695133 4756 webserver.cc:460] Webserver started at http://127.3.220.62:43523/ using document root <none> and password file <none>
I20260809 01:57:17.695698 4756 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:17.695763 4756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:17.695988 4756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:17.698096 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/data/instance:
uuid: "9a75f9cd035d48d7998740c77177a61e"
format_stamp: "Formatted at 2026-08-09 01:57:17 on dist-test-slave-gkw7"
server_key: "acefde75453242e901e6c80ff9db39a9"
server_key_iv: "e62093c12513a6492cef10eb190c4016"
server_key_version: "encryptionkey@0"
I20260809 01:57:17.698689 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/wal/instance:
uuid: "9a75f9cd035d48d7998740c77177a61e"
format_stamp: "Formatted at 2026-08-09 01:57:17 on dist-test-slave-gkw7"
server_key: "acefde75453242e901e6c80ff9db39a9"
server_key_iv: "e62093c12513a6492cef10eb190c4016"
server_key_version: "encryptionkey@0"
I20260809 01:57:17.702409 4756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 01:57:17.704829 4775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:17.706133 4756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 01:57:17.706282 4756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/wal
uuid: "9a75f9cd035d48d7998740c77177a61e"
format_stamp: "Formatted at 2026-08-09 01:57:17 on dist-test-slave-gkw7"
server_key: "acefde75453242e901e6c80ff9db39a9"
server_key_iv: "e62093c12513a6492cef10eb190c4016"
server_key_version: "encryptionkey@0"
I20260809 01:57:17.706395 4756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:17.729732 4756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:17.733175 4756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:17.733404 4756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:17.740713 4756 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.62:41169
I20260809 01:57:17.741034 4851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.62:41169 every 8 connection(s)
I20260809 01:57:17.741748 4756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/data/info.pb
I20260809 01:57:17.744657 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:17.748488 4852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:17.750645 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4756
I20260809 01:57:17.750779 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-0/wal/instance
I20260809 01:57:17.751125 3952 external_mini_cluster.cc:1428] Setting key 86c5f45f6f1868c32bcce225d3f11383
Aug 09 01:57:17 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.220.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.61@KRBTEST.COM" created.
I20260809 01:57:17.763652 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.750691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.3.220.62:41169)
0809 01:57:17.751239 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:17.751258 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:17.751450 (+ 192us) server_negotiation.cc:407] Connection header received
0809 01:57:17.752469 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:17.752473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:17.752689 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:17.752892 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:17.753810 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.754670 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.755678 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.756051 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.756225 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:17.758615 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:17.758621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:17.758636 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:17.758666 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:17.760453 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:17.760925 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:17.760930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:17.760931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:17.761039 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:17.761325 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:17.761327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:17.761328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:17.761808 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:17.761932 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:17.762426 (+ 494us) server_negotiation.cc:299] Negotiation successful
0809 01:57:17.762721 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":414,"spinlock_wait_cycles":55680,"thread_start_us":183,"threads_started":1}
I20260809 01:57:17.763654 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.750693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41169 (local address 127.0.0.1:49898)
0809 01:57:17.751301 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.751347 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:17.752144 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:17.752890 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:17.752895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:17.753338 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:17.753644 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:17.753660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.754827 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.754836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:17.755551 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:17.755560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.756208 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.756212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:17.756332 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:17.756950 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:17.756967 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:17.758442 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:17.760561 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:17.760570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:17.760578 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:17.760821 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:17.761125 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:17.761127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:17.761128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:17.761241 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:17.762018 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:17.762023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:17.762183 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 01:57:17.762355 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 01:57:17.762806 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":175,"thread_start_us":149,"threads_started":1}
I20260809 01:57:17.766139 4852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:17.766868 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.766621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53878)
0809 01:57:17.766740 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.766818 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 01:57:17.767059 4776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
I20260809 01:57:17.767908 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.767704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53894)
0809 01:57:17.767808 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.767853 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 01:57:17.768074 4852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37269: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab.
Entry for principal kudu/127.3.220.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab.
I20260809 01:57:17.781404 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.61:37269
--webserver_interface=127.3.220.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab
--principal=kudu/127.3.220.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:41169,127.3.220.61:37269,127.3.220.60:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:57:17.817296 4852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } attempt: 1
I20260809 01:57:17.818287 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.817934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53904)
0809 01:57:17.818114 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.818190 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 01:57:17.819294 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.819053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53918)
0809 01:57:17.819172 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.819227 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 01:57:17.819437 4852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37269: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
I20260809 01:57:17.887589 4852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } attempt: 2
I20260809 01:57:17.888652 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.888219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53922)
0809 01:57:17.888461 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.888538 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 01:57:17.889564 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.889287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53936)
0809 01:57:17.889408 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.889463 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 01:57:17.889745 4852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.61:37269: Network error: Client connection negotiation failed: client connection to 127.3.220.61:37269: connect: Connection refused (error 111)
W20260809 01:57:17.897568 4861 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 01:57:17.897886 4861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:17.897936 4861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:17.898027 4861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:17.902110 4861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:17.902175 4861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:17.902195 4861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:17.902212 4861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:17.902228 4861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:17.908114 4861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:41169,127.3.220.61:37269,127.3.220.60:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.61.keytab
--principal=kudu/127.3.220.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.61:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.220.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:17.909298 4861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:17.910409 4861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:17.919452 4872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:17.919446 4870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:17.919602 4869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:17.919730 4861 server_base.cc:1034] running on GCE node
I20260809 01:57:17.920423 4861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:17.921646 4861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:17.922839 4861 hybrid_clock.cc:648] HybridClock initialized: now 1786240637922819 us; error 31 us; skew 500 ppm
Aug 09 01:57:17 dist-test-slave-gkw7 krb5kdc[4737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:17.925839 4861 init.cc:377] Logged in from keytab as kudu/127.3.220.61@KRBTEST.COM (short username kudu)
I20260809 01:57:17.926743 4861 webserver.cc:460] Webserver started at http://127.3.220.61:46687/ using document root <none> and password file <none>
I20260809 01:57:17.927350 4861 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:17.927446 4861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:17.927707 4861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:17.929881 4861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/data/instance:
uuid: "2cb6e54d047647009cbd8ddbd62a0879"
format_stamp: "Formatted at 2026-08-09 01:57:17 on dist-test-slave-gkw7"
server_key: "4faf801f52dc384290dfc1978becbe17"
server_key_iv: "f300e7bd6aad60fac3a4035cae3ef693"
server_key_version: "encryptionkey@0"
I20260809 01:57:17.930467 4861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/wal/instance:
uuid: "2cb6e54d047647009cbd8ddbd62a0879"
format_stamp: "Formatted at 2026-08-09 01:57:17 on dist-test-slave-gkw7"
server_key: "4faf801f52dc384290dfc1978becbe17"
server_key_iv: "f300e7bd6aad60fac3a4035cae3ef693"
server_key_version: "encryptionkey@0"
I20260809 01:57:17.934508 4861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 01:57:17.937091 4884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:17.938406 4861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 01:57:17.938565 4861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/wal
uuid: "2cb6e54d047647009cbd8ddbd62a0879"
format_stamp: "Formatted at 2026-08-09 01:57:17 on dist-test-slave-gkw7"
server_key: "4faf801f52dc384290dfc1978becbe17"
server_key_iv: "f300e7bd6aad60fac3a4035cae3ef693"
server_key_version: "encryptionkey@0"
I20260809 01:57:17.938694 4861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:17.953178 4861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:17.956312 4861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:17.956533 4861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:17.963595 4861 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.61:37269
I20260809 01:57:17.963898 4953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.61:37269 every 8 connection(s)
I20260809 01:57:17.964591 4861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/data/info.pb
I20260809 01:57:17.967455 4954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:17.968519 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4861
I20260809 01:57:17.968669 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-1/wal/instance
I20260809 01:57:17.968990 3952 external_mini_cluster.cc:1428] Setting key 6585aa3578f61268baf5ebbda1c6943d
I20260809 01:57:17.971326 4954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.220.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.220.60@KRBTEST.COM" created.
I20260809 01:57:17.980911 4852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } attempt: 3
Aug 09 01:57:17 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:57:17.986166 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.973261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.3.220.62:41169)
0809 01:57:17.973403 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:17.973407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:17.974573 (+ 1166us) server_negotiation.cc:407] Connection header received
0809 01:57:17.975583 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:17.975587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:17.975655 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:17.975773 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:17.976928 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.977469 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.978707 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.978957 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.979072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:17.981810 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:17.981814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:17.981817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:17.981861 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:17.983715 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:17.984364 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:17.984369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:17.984371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:17.984461 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:17.984925 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:17.984928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:17.984930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:17.985149 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:17.985258 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:17.985871 (+ 613us) server_negotiation.cc:299] Negotiation successful
0809 01:57:17.985990 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:57:17.987054 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.973788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41169 (local address 127.0.0.1:49902)
0809 01:57:17.974403 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.974446 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:17.975365 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:17.975947 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:17.975954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:17.976320 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:17.976780 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:17.976793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.977639 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.977642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:17.978548 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:17.978583 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.979115 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.979119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:17.979292 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:17.979929 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:17.979946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:17.981618 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:17.983842 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:17.983851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:17.983868 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:17.984255 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:17.984567 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:17.984573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:17.984578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:17.984784 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:17.985259 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:17.985266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:17.985611 (+ 345us) client_negotiation.cc:769] Sending connection context
0809 01:57:17.985815 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 01:57:17.986128 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":161,"threads_started":1}
Aug 09 01:57:17 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:57:17.989627 4954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:17.992668 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.981369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53938)
0809 01:57:17.981580 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.981596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:17.981686 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:17.982460 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:17.982466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:17.982701 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:17.982837 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:17.982843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.984202 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.984206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:17.984945 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:17.984954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.985942 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.985947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:17.986080 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:17.986599 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:17.986636 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:17.988056 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:17.990955 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:17.990959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:17.990964 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:17.991202 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:17.991531 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:17.991534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:17.991536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:17.991623 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:17.992121 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:17.992124 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:17.992259 (+ 135us) client_negotiation.cc:769] Sending connection context
0809 01:57:17.992374 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 01:57:17.992518 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260809 01:57:17.992741 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.981507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.3.220.61:37269)
0809 01:57:17.981983 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:17.982001 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:17.982028 (+ 27us) server_negotiation.cc:407] Connection header received
0809 01:57:17.982076 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:17.982085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:17.982296 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:17.982443 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:17.982979 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.984082 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.985072 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.985787 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.987322 (+ 1535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:17.988195 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:17.988203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:17.988216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:17.988239 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:17.990861 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:17.991320 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:17.991325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:17.991327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:17.991423 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:17.991726 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:17.991730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:17.991731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:17.991995 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:17.992120 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:17.992482 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 01:57:17.992590 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":128,"threads_started":1}
I20260809 01:57:17.994829 4852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:17.995571 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.995308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34294)
0809 01:57:17.995439 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.995494 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
Aug 09 01:57:17 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:57:17.996454 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.996119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34298)
0809 01:57:17.996327 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.996383 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260809 01:57:17.996639 4852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:42067: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.220.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab.
Entry for principal kudu/127.3.220.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab.
I20260809 01:57:17.999723 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.990344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.3.220.61:37269)
0809 01:57:17.990639 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:17.990645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:17.990656 (+ 11us) server_negotiation.cc:407] Connection header received
0809 01:57:17.990692 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:17.990694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:17.990725 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:17.990827 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:17.991309 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.992160 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.993209 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.993920 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.994009 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:17.996401 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:17.996405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:17.996407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:17.996430 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:17.998012 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:17.998471 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:17.998473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:17.998475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:17.998552 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:17.998849 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:17.998852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:17.998853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:17.999020 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:17.999098 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:17.999399 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 01:57:17.999544 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260809 01:57:18.000381 3952 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
/tmp/dist-test-taskIuGTYa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.60:42067
--webserver_interface=127.3.220.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab
--principal=kudu/127.3.220.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.220.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.220.62:41169,127.3.220.61:37269,127.3.220.60:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 01:57:18.001627 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:17.990145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53940)
0809 01:57:17.990339 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:17.990351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:17.990414 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:17.990855 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:17.990858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:17.991028 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:17.991138 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:17.991144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.992276 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.992280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:17.993079 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:17.993088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:17.994047 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:17.994050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:17.994146 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:17.994682 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:17.994704 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:17.996239 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:17.998116 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:17.998119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:17.998120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:17.998380 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:17.998637 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:17.998639 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:17.998640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:17.998730 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:17.999105 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:17.999107 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:17.999205 (+ 98us) client_negotiation.cc:769] Sending connection context
0809 01:57:17.999333 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 01:57:17.999475 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260809 01:57:18.002596 4954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:18.003315 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.003062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34308)
0809 01:57:18.003175 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.003260 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 01:57:18.003538 4887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
I20260809 01:57:18.004403 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.004193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34318)
0809 01:57:18.004302 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.004353 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 01:57:18.004580 4954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:42067: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
I20260809 01:57:18.027823 4852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } attempt: 1
I20260809 01:57:18.028793 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.028413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34326)
0809 01:57:18.028603 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.028684 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 01:57:18.029608 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.029344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34328)
0809 01:57:18.029457 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.029515 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 01:57:18.029785 4852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:42067: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
I20260809 01:57:18.053916 4954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } attempt: 1
I20260809 01:57:18.054926 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.054551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34344)
0809 01:57:18.054747 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.054823 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 01:57:18.055855 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.055579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34348)
0809 01:57:18.055730 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.055787 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 01:57:18.056107 4954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:42067: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
I20260809 01:57:18.104995 4852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } attempt: 2
I20260809 01:57:18.105993 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.105626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34354)
0809 01:57:18.105820 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.105895 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 01:57:18.106839 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.106586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34370)
0809 01:57:18.106719 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.106777 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 01:57:18.106986 4852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:42067: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
W20260809 01:57:18.117280 4962 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 01:57:18.117638 4962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 01:57:18.117691 4962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 01:57:18.117774 4962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 01:57:18.121789 4962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 01:57:18.121858 4962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 01:57:18.121902 4962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 01:57:18.121940 4962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 01:57:18.121975 4962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 01:57:18.124290 4954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } attempt: 2
I20260809 01:57:18.125126 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.124806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34376)
0809 01:57:18.124969 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.125036 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 01:57:18.125916 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.125692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34378)
0809 01:57:18.125800 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.125851 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 01:57:18.126050 4954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.220.60:42067: Network error: Client connection negotiation failed: client connection to 127.3.220.60:42067: connect: Connection refused (error 111)
I20260809 01:57:18.127873 4962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.220.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.220.62:41169,127.3.220.61:37269,127.3.220.60:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/krb5kdc/kudu_127.3.220.60.keytab
--principal=kudu/127.3.220.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.220.60:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.220.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 01:21:56 UTC on ccycloud.quasar-xrrvkm.root.comops.site
build id 4575763
I20260809 01:57:18.128876 4962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 01:57:18.129914 4962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 01:57:18.138368 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 01:57:18.138398 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:18.138500 4962 server_base.cc:1034] running on GCE node
W20260809 01:57:18.138368 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 01:57:18.139281 4962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 01:57:18.140393 4962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 01:57:18.141589 4962 hybrid_clock.cc:648] HybridClock initialized: now 1786240638141573 us; error 27 us; skew 500 ppm
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 01:57:18.144721 4962 init.cc:377] Logged in from keytab as kudu/127.3.220.60@KRBTEST.COM (short username kudu)
I20260809 01:57:18.145732 4962 webserver.cc:460] Webserver started at http://127.3.220.60:45357/ using document root <none> and password file <none>
I20260809 01:57:18.146320 4962 fs_manager.cc:359] Metadata directory not provided
I20260809 01:57:18.146414 4962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 01:57:18.146654 4962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 01:57:18.148772 4962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/data/instance:
uuid: "8ac0a7ef275e42289ea4f78c8a97d89d"
format_stamp: "Formatted at 2026-08-09 01:57:18 on dist-test-slave-gkw7"
server_key: "d4bb491a6674117d315eeac199128fad"
server_key_iv: "85baaa0ecc5912eeeadd556933b63106"
server_key_version: "encryptionkey@0"
I20260809 01:57:18.149354 4962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/wal/instance:
uuid: "8ac0a7ef275e42289ea4f78c8a97d89d"
format_stamp: "Formatted at 2026-08-09 01:57:18 on dist-test-slave-gkw7"
server_key: "d4bb491a6674117d315eeac199128fad"
server_key_iv: "85baaa0ecc5912eeeadd556933b63106"
server_key_version: "encryptionkey@0"
I20260809 01:57:18.153136 4962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260809 01:57:18.155653 4984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 01:57:18.156857 4962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 01:57:18.157006 4962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/data,/tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/wal
uuid: "8ac0a7ef275e42289ea4f78c8a97d89d"
format_stamp: "Formatted at 2026-08-09 01:57:18 on dist-test-slave-gkw7"
server_key: "d4bb491a6674117d315eeac199128fad"
server_key_iv: "85baaa0ecc5912eeeadd556933b63106"
server_key_version: "encryptionkey@0"
I20260809 01:57:18.157119 4962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 01:57:18.171525 4962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 01:57:18.175076 4962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 01:57:18.175287 4962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 01:57:18.182219 4962 rpc_server.cc:307] RPC server started. Bound to: 127.3.220.60:42067
I20260809 01:57:18.182509 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.220.60:42067 every 8 connection(s)
I20260809 01:57:18.183240 4962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/data/info.pb
I20260809 01:57:18.186101 5065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 01:57:18.188349 3952 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu as pid 4962
I20260809 01:57:18.188513 3952 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIuGTYa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786240338176169-3952-0/minicluster-data/master-2/wal/instance
I20260809 01:57:18.188827 3952 external_mini_cluster.cc:1428] Setting key fe9163304c5e3b571b74c0ebb338a587
I20260809 01:57:18.190253 5065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
I20260809 01:57:18.203461 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.192193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.3.220.62:41169)
0809 01:57:18.192809 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.192813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.192824 (+ 11us) server_negotiation.cc:407] Connection header received
0809 01:57:18.192860 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.192863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.192898 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.192961 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.193831 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.194587 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.195971 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.196286 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.196410 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.199098 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.199102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.199107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.199136 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.200848 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.201434 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.201437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.201439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.201586 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.201979 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.201983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.201985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.202339 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.202462 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.203053 (+ 591us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.203247 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526}
I20260809 01:57:18.204279 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.190312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.3.220.60:42067)
0809 01:57:18.190736 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.190754 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.190798 (+ 44us) server_negotiation.cc:407] Connection header received
0809 01:57:18.191608 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.191629 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.191994 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.192346 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.193235 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.194627 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.195828 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.196334 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.196474 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.199627 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.199635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.199650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.199679 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.201580 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.202094 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.202098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.202100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.202166 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.202477 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.202480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.202482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.202776 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.202968 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.203276 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.203393 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260809 01:57:18.204272 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.192100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.62:41169 (local address 127.0.0.1:49910)
0809 01:57:18.192489 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.192535 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:18.192632 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:18.193072 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:18.193077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:18.193465 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:18.193691 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.193702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.194743 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.194746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.195868 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.195877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.196494 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.196498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.196616 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.197252 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:18.197273 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:18.198945 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:18.200966 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.200974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.200985 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.201308 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.201730 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.201733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.201734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.201875 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.202477 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:18.202483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:18.202797 (+ 314us) client_negotiation.cc:769] Sending connection context
0809 01:57:18.202968 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 01:57:18.203286 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"spinlock_wait_cycles":2944,"thread_start_us":138,"threads_started":1}
I20260809 01:57:18.205636 5065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:18.206190 4852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } attempt: 3
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.62@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.62@KRBTEST.COM
I20260809 01:57:18.216940 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.206534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34406)
0809 01:57:18.206674 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.206691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:18.206772 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:18.207084 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:18.207088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:18.207283 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:18.207388 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.207396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.208523 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.208526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.209291 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.209301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.210014 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.210018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.210096 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.210674 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:18.210688 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:18.213132 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:18.215277 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.215282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.215284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.215534 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.215820 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.215823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.215825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.215916 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.216352 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:18.216355 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:18.216474 (+ 119us) client_negotiation.cc:769] Sending connection context
0809 01:57:18.216592 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 01:57:18.216756 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 01:57:18.216965 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.206755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.3.220.60:42067)
0809 01:57:18.206849 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.206853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.206865 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:57:18.206898 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.206901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.206946 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.207501 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.207593 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.208401 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.209461 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.209887 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.210306 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.213277 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.213282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.213285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.213312 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.215157 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.215637 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.215640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.215642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.215727 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.216008 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.216013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.216015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.216208 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.216366 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.216654 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.216782 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":3456}
I20260809 01:57:18.217203 4954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } attempt: 3
I20260809 01:57:18.217905 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.207287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.3.220.62:41169)
0809 01:57:18.207382 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.207386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.207400 (+ 14us) server_negotiation.cc:407] Connection header received
0809 01:57:18.207436 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.207440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.207487 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.207559 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.208288 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.208866 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.210003 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.210316 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.210390 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.213342 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.213347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.213350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.213376 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.215505 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.216082 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.216085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.216097 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.216195 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.216522 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.216526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.216527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.216731 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.216959 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.217640 (+ 681us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.217759 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 01:57:18.218030 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.206350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.3.220.61:37269)
0809 01:57:18.206510 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.206515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.206733 (+ 218us) server_negotiation.cc:407] Connection header received
0809 01:57:18.206956 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.206966 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.207014 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.207142 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.207881 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.208611 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.209574 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.209842 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.209976 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.213777 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.213782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.213785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.213823 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.215709 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.216241 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.216246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.216250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.216345 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.216715 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.216720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.216722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.216941 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.217028 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.217716 (+ 688us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.217852 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 01:57:18.218789 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.206216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.61:37269 (local address 127.0.0.1:53942)
0809 01:57:18.206647 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.206663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:18.206801 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:18.207309 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:18.207315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:18.207518 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:18.207707 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.207718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.208736 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.208740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.209359 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.209394 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.209972 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.209976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.210080 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.210794 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:18.210815 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:18.213563 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:18.215846 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.215852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.215857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.216113 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.216474 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.216477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.216479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.216582 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.217348 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:18.217351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:18.217454 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 01:57:18.218431 (+ 977us) client_negotiation.cc:240] Negotiation successful
0809 01:57:18.218580 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":4}
I20260809 01:57:18.219574 5065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } has no permanent_uuid. Determining permanent_uuid...
I20260809 01:57:18.221061 4852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e: Bootstrap starting.
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.61@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
I20260809 01:57:18.224154 4852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:18.225149 4852 log.cc:824] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e: Log is configured to *not* fsync() on all Append() calls
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.220.60@KRBTEST.COM for kudu/127.3.220.60@KRBTEST.COM
Aug 09 01:57:18 dist-test-slave-gkw7 krb5kdc[4737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786240637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.220.61@KRBTEST.COM
I20260809 01:57:18.228250 4852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e: No bootstrap required, opened a new log
I20260809 01:57:18.228335 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.218108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.3.220.60:42067)
0809 01:57:18.218214 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.218218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.218229 (+ 11us) server_negotiation.cc:407] Connection header received
0809 01:57:18.218260 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.218262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.218298 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.218834 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.218920 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.219570 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.220638 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.221008 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.223400 (+ 2392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.224398 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.224404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.224409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.224432 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.226346 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.226907 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.226914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.226919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.227022 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.227387 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.227391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.227393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.227627 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.227724 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.228060 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.228179 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":47}
I20260809 01:57:18.228335 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.217613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34420)
0809 01:57:18.217873 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.217886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:18.217964 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:18.218415 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:18.218418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:18.218612 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:18.218722 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.218729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.219706 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.219712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.220491 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.220503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.221144 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.221147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.221261 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.221761 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:18.221775 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:18.223259 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:18.226460 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.226464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.226467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.226768 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.227150 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.227154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.227157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.227259 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.227781 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:18.227784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:18.227875 (+ 91us) client_negotiation.cc:769] Sending connection context
0809 01:57:18.228003 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 01:57:18.228149 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":28}
I20260809 01:57:18.231611 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.220034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.220.60:42067 (local address 127.0.0.1:34422)
0809 01:57:18.220690 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0809 01:57:18.220706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 01:57:18.220786 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 01:57:18.221389 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 01:57:18.221392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 01:57:18.221600 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0809 01:57:18.221692 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.221700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.223321 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.223324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.224181 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 01:57:18.224195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.224882 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.224889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 01:57:18.224998 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.225453 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 01:57:18.225470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 01:57:18.227592 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 01:57:18.230175 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.230177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.230179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.230402 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.230664 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 01:57:18.230667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 01:57:18.230669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 01:57:18.230767 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 01:57:18.231148 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 01:57:18.231150 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 01:57:18.231224 (+ 74us) client_negotiation.cc:769] Sending connection context
0809 01:57:18.231314 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 01:57:18.231437 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":16,"thread_start_us":115,"threads_started":1}
I20260809 01:57:18.232004 4954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879: Bootstrap starting.
I20260809 01:57:18.232470 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.220383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.3.220.60:42067)
0809 01:57:18.221080 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.221092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.221107 (+ 15us) server_negotiation.cc:407] Connection header received
0809 01:57:18.221144 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.221149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.221259 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.221808 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.221910 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.223194 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.224337 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.224725 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.224874 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.227750 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.227754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.227757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.227780 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.230059 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.230499 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.230502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.230504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.230576 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.230855 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.230858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.230860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.231035 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.231118 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.232182 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.232300 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":104,"threads_started":1}
I20260809 01:57:18.232923 4852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } }
I20260809 01:57:18.233168 4852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a75f9cd035d48d7998740c77177a61e, State: Initialized, Role: FOLLOWER
I20260809 01:57:18.233520 4852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } }
I20260809 01:57:18.234210 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.219821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.3.220.61:37269)
0809 01:57:18.219909 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.219913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.220089 (+ 176us) server_negotiation.cc:407] Connection header received
0809 01:57:18.220258 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.220261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.220299 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.220342 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.221132 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.221900 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.223023 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.223359 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.223434 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.230165 (+ 6731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.230169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.230171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.230197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.232013 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.232576 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.232580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.232582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.232689 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.233043 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.233046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.233048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.233270 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.233432 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.233932 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.234049 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260809 01:57:18.235031 4785 tablet.cc:2369] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 01:57:18.235062 4954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:18.235493 5065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d: Bootstrap starting.
I20260809 01:57:18.235792 4954 log.cc:824] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:18.235989 4852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:18.238255 5065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d: Neither blocks nor log segments found. Creating new log.
I20260809 01:57:18.238916 3952 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 01:57:18.239112 5065 log.cc:824] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d: Log is configured to *not* fsync() on all Append() calls
I20260809 01:57:18.239413 4954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879: No bootstrap required, opened a new log
I20260809 01:57:18.240139 4809 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9a75f9cd035d48d7998740c77177a61e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49918
I20260809 01:57:18.240375 4809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 01:57:18.240507 4809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:18.241968 4809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } }
I20260809 01:57:18.242988 4809 leader_election.cc:288] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [CANDIDATE]: Term 1 election: Requested vote from peers 2cb6e54d047647009cbd8ddbd62a0879 (127.3.220.61:37269), 8ac0a7ef275e42289ea4f78c8a97d89d (127.3.220.60:42067)
I20260809 01:57:18.243973 4954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } }
I20260809 01:57:18.244247 4954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb6e54d047647009cbd8ddbd62a0879, State: Initialized, Role: FOLLOWER
I20260809 01:57:18.244594 4954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } }
I20260809 01:57:18.245018 5065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d: No bootstrap required, opened a new log
I20260809 01:57:18.246009 4954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:18.244395 4756 master_runner.cc:186] Error getting master registration for 127.3.220.61:37269: OK, instance_id { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" instance_seqno: 1786240637961609 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 01:57:18.248255 4756 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:57:18.248374 4920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a75f9cd035d48d7998740c77177a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2cb6e54d047647009cbd8ddbd62a0879"
I20260809 01:57:18.248646 4920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:18.249135 5065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } }
I20260809 01:57:18.249389 5065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac0a7ef275e42289ea4f78c8a97d89d, State: Initialized, Role: FOLLOWER
I20260809 01:57:18.249753 5065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } }
I20260809 01:57:18.250208 4920 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a75f9cd035d48d7998740c77177a61e in term 1.
I20260809 01:57:18.251111 5065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 01:57:18.251278 4776 leader_election.cc:302] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb6e54d047647009cbd8ddbd62a0879, 9a75f9cd035d48d7998740c77177a61e; no voters:
I20260809 01:57:18.253006 5022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a75f9cd035d48d7998740c77177a61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d"
W20260809 01:57:18.253322 5089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:57:18.253414 5089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 01:57:18.253647 5111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:57:18.253731 5111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:57:18.253662 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.253790 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:18.253895 5076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 01:57:18.253978 5076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [term 1 LEADER]: Becoming Leader. State: Replica: 9a75f9cd035d48d7998740c77177a61e, State: Running, Role: LEADER
I20260809 01:57:18.254141 5112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.254287 5112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:18.254316 5076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } }
I20260809 01:57:18.256070 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.256099 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a75f9cd035d48d7998740c77177a61e. Latest consensus state: current_term: 1 leader_uuid: "9a75f9cd035d48d7998740c77177a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.256225 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:18.256230 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:18.256390 5022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [term 0 FOLLOWER]: Advancing to term 1
I20260809 01:57:18.258279 5022 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a75f9cd035d48d7998740c77177a61e in term 1.
I20260809 01:57:18.266374 4861 master_runner.cc:263] No leader master found from master 2cb6e54d047647009cbd8ddbd62a0879
I20260809 01:57:18.266472 4861 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 01:57:18.266717 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.256310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.3.220.61:37269)
0809 01:57:18.256443 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.256449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.257289 (+ 840us) server_negotiation.cc:407] Connection header received
0809 01:57:18.257487 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.257490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.257567 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.257622 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.258661 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.259519 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.260672 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.260961 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.261039 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.263028 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.263032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.263036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.263070 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.264820 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.265323 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.265326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.265328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.265411 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.265737 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.265741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.265742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.265956 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.266055 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.266424 (+ 369us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.266546 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 01:57:18.272452 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.256689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.3.220.62:41169)
0809 01:57:18.256837 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.256842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.257701 (+ 859us) server_negotiation.cc:407] Connection header received
0809 01:57:18.257915 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.257919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.257968 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.258026 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.259617 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.260375 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.265855 (+ 5480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.266221 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.266310 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.268271 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.268276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.268279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.268308 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.270202 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.270829 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.270834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.270836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.270937 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.271317 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.271321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.271323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.271566 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.271697 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.272129 (+ 432us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.272271 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 01:57:18.275225 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:18.257737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.3.220.60:42067)
0809 01:57:18.257860 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.257867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.258578 (+ 711us) server_negotiation.cc:407] Connection header received
0809 01:57:18.258751 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.258755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.258804 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.258877 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.266609 (+ 7732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.267354 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.269169 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:18.269498 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:18.269635 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:18.271305 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 01:57:18.271309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 01:57:18.271311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 01:57:18.271336 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 01:57:18.273057 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.273606 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.273611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.273613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.273717 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 01:57:18.274104 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 01:57:18.274108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 01:57:18.274113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 01:57:18.274352 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 01:57:18.274487 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:18.274900 (+ 413us) server_negotiation.cc:299] Negotiation successful
0809 01:57:18.275040 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 01:57:18.277810 5140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 01:57:18.278635 5140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260809 01:57:18.279254 5143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 01:57:18.279356 5143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 01:57:18.289803 4920 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [term 1 FOLLOWER]: Refusing update from remote peer 9a75f9cd035d48d7998740c77177a61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:57:18.289839 5022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [term 1 FOLLOWER]: Refusing update from remote peer 9a75f9cd035d48d7998740c77177a61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 01:57:18.290469 5113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:57:18.290650 5076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 01:57:18.294790 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a75f9cd035d48d7998740c77177a61e. Latest consensus state: current_term: 1 leader_uuid: "9a75f9cd035d48d7998740c77177a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.294935 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:18.295998 5112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a75f9cd035d48d7998740c77177a61e. Latest consensus state: current_term: 1 leader_uuid: "9a75f9cd035d48d7998740c77177a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.296119 5112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:18.300124 5113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a75f9cd035d48d7998740c77177a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.300249 5113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:18.301473 5112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a75f9cd035d48d7998740c77177a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.301610 5112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:18.301914 5152 mvcc.cc:205] Tried to move back new op lower bound from 7316441654428196864 to 7316441654293090304. Current Snapshot: MvccSnapshot[applied={T|T < 7316441654428196864}]
I20260809 01:57:18.302004 5145 mvcc.cc:205] Tried to move back new op lower bound from 7316441654428196864 to 7316441654293090304. Current Snapshot: MvccSnapshot[applied={T|T < 7316441654428196864}]
I20260809 01:57:18.304725 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a75f9cd035d48d7998740c77177a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.304837 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e [sys.catalog]: This master's current role is: LEADER
I20260809 01:57:18.305927 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a75f9cd035d48d7998740c77177a61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a75f9cd035d48d7998740c77177a61e" member_type: VOTER last_known_addr { host: "127.3.220.62" port: 41169 } } peers { permanent_uuid: "2cb6e54d047647009cbd8ddbd62a0879" member_type: VOTER last_known_addr { host: "127.3.220.61" port: 37269 } } peers { permanent_uuid: "8ac0a7ef275e42289ea4f78c8a97d89d" member_type: VOTER last_known_addr { host: "127.3.220.60" port: 42067 } } }
I20260809 01:57:18.306027 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 01:57:18.308869 5148 mvcc.cc:205] Tried to move back new op lower bound from 7316441654428196864 to 7316441654293090304. Current Snapshot: MvccSnapshot[applied={T|T < 7316441654428196864}]
I20260809 01:57:18.312287 5140 catalog_manager.cc:1347] Generated new cluster ID: a5d9ea934d134f47a0c1ce6c404267df
I20260809 01:57:18.312355 5140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 01:57:18.326206 5140 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 01:57:18.327566 5140 catalog_manager.cc:1504] Loading token signing keys...
I20260809 01:57:18.341043 5140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a75f9cd035d48d7998740c77177a61e: Generated new TSK 0
I20260809 01:57:18.342100 5140 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 01:57:18.376842 4857 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.373068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.3.220.62:41169)
0809 01:57:18.373219 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.373225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.375198 (+ 1973us) server_negotiation.cc:407] Connection header received
0809 01:57:18.375773 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.375778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.375840 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.375902 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.376674 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49924: BlockingRecv error: recv got EOF from 127.0.0.1:49924 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 01:57:18.378415 5074 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.374007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.3.220.60:42067)
0809 01:57:18.374139 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.374145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.377296 (+ 3151us) server_negotiation.cc:407] Connection header received
0809 01:57:18.377497 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.377501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.377581 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.377653 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.378278 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34436: BlockingRecv error: recv got EOF from 127.0.0.1:34436 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260809 01:57:18.380224 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.374790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.3.220.61:37269)
0809 01:57:18.374920 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.374925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.379008 (+ 4083us) server_negotiation.cc:407] Connection header received
0809 01:57:18.379369 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.379373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.379430 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.379491 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.380094 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53984: BlockingRecv error: recv got EOF from 127.0.0.1:53984 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 01:57:18.385387 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.383753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.3.220.61:37269)
0809 01:57:18.383865 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.383869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.384331 (+ 462us) server_negotiation.cc:407] Connection header received
0809 01:57:18.384577 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.384581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.384680 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.384738 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.385279 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53992: BlockingRecv error: recv got EOF from 127.0.0.1:53992 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 01:57:18.390924 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.389337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.3.220.61:37269)
0809 01:57:18.389443 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.389448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.389983 (+ 535us) server_negotiation.cc:407] Connection header received
0809 01:57:18.390183 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.390187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.390239 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.390294 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.390823 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53994: BlockingRecv error: recv got EOF from 127.0.0.1:53994 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260809 01:57:18.396641 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.395251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.3.220.61:37269)
0809 01:57:18.395348 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.395352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.395810 (+ 458us) server_negotiation.cc:407] Connection header received
0809 01:57:18.395995 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.395998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.396043 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.396091 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.396550 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54000: BlockingRecv error: recv got EOF from 127.0.0.1:54000 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 01:57:18.404579 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.403021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.3.220.61:37269)
0809 01:57:18.403161 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.403166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.403635 (+ 469us) server_negotiation.cc:407] Connection header received
0809 01:57:18.403861 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.403864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.403914 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.403965 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.404476 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54016: BlockingRecv error: recv got EOF from 127.0.0.1:54016 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 01:57:18.416585 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.415002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.3.220.61:37269)
0809 01:57:18.415113 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.415118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.415615 (+ 497us) server_negotiation.cc:407] Connection header received
0809 01:57:18.415831 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.415835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.415889 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.415940 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.416483 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54032: BlockingRecv error: recv got EOF from 127.0.0.1:54032 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260809 01:57:18.437450 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.435526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.3.220.61:37269)
0809 01:57:18.435669 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.435675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.436298 (+ 623us) server_negotiation.cc:407] Connection header received
0809 01:57:18.436542 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.436546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.436610 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.436683 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.437321 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54046: BlockingRecv error: recv got EOF from 127.0.0.1:54046 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 01:57:18.474231 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.472431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.3.220.61:37269)
0809 01:57:18.472561 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.472566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.473161 (+ 595us) server_negotiation.cc:407] Connection header received
0809 01:57:18.473374 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.473377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.473431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.473483 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.474106 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54048: BlockingRecv error: recv got EOF from 127.0.0.1:54048 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 01:57:18.542332 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.540955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.3.220.61:37269)
0809 01:57:18.541155 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.541160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.541220 (+ 60us) server_negotiation.cc:407] Connection header received
0809 01:57:18.541354 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.541357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.541413 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.541496 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.542163 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54064: BlockingRecv error: recv got EOF from 127.0.0.1:54064 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260809 01:57:18.674329 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.672876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.3.220.61:37269)
0809 01:57:18.673055 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.673061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.673272 (+ 211us) server_negotiation.cc:407] Connection header received
0809 01:57:18.673476 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.673480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.673565 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.673677 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.674195 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54074: BlockingRecv error: recv got EOF from 127.0.0.1:54074 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 01:57:18.934365 4961 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 01:57:18.932974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.3.220.61:37269)
0809 01:57:18.933169 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:18.933177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:18.933204 (+ 27us) server_negotiation.cc:407] Connection header received
0809 01:57:18.933444 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:18.933455 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:18.933509 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:18.933644 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 01:57:18.934208 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54076: BlockingRecv error: recv got EOF from 127.0.0.1:54076 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 01:57:19.255054 5111 catalog_manager.cc:1259] Loaded cluster ID: a5d9ea934d134f47a0c1ce6c404267df
I20260809 01:57:19.255154 5111 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879: loading cluster ID for follower catalog manager: success
I20260809 01:57:19.256775 5111 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879: acquiring CA information for follower catalog manager: success
I20260809 01:57:19.257946 5111 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cb6e54d047647009cbd8ddbd62a0879: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:57:19.280645 5143 catalog_manager.cc:1259] Loaded cluster ID: a5d9ea934d134f47a0c1ce6c404267df
I20260809 01:57:19.280725 5143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d: loading cluster ID for follower catalog manager: success
I20260809 01:57:19.282474 5143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d: acquiring CA information for follower catalog manager: success
I20260809 01:57:19.283595 5143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ac0a7ef275e42289ea4f78c8a97d89d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 01:57:19.453084 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:19.448886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.3.220.61:37269)
0809 01:57:19.449132 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:19.449136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:19.449225 (+ 89us) server_negotiation.cc:407] Connection header received
0809 01:57:19.449360 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:19.449362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:19.449493 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:19.449634 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:19.449962 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.450494 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.451447 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.451678 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.451776 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:19.452173 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:19.452639 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:19.452735 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:19.452821 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 01:57:19.452873 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260809 01:57:19.459944 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:19.456302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.3.220.60:42067)
0809 01:57:19.456547 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:19.456550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:19.456562 (+ 12us) server_negotiation.cc:407] Connection header received
0809 01:57:19.456639 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:19.456642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:19.456749 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:19.456858 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:19.457081 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.457676 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.458493 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.458705 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.458785 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:19.459101 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:19.459490 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:19.459582 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:19.459716 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 01:57:19.459817 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260809 01:57:19.467010 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:19.463110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.3.220.61:37269)
0809 01:57:19.463324 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:19.463338 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:19.463597 (+ 259us) server_negotiation.cc:407] Connection header received
0809 01:57:19.463708 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:19.463711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:19.463769 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:19.463841 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:19.464075 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.464883 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.465928 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.466152 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.466257 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:19.466536 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:19.466607 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:19.466674 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:19.466779 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 01:57:19.466858 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260809 01:57:19.467742 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:19.463425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.3.220.60:42067)
0809 01:57:19.463584 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:19.463589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:19.463683 (+ 94us) server_negotiation.cc:407] Connection header received
0809 01:57:19.463785 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:19.463790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:19.463849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:19.463909 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:19.464275 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.465004 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.466477 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.466801 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.466886 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:19.467296 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:19.467372 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:19.467428 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:19.467587 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 01:57:19.467640 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 01:57:19.468333 5267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 01:57:19.463033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.3.220.62:41169)
0809 01:57:19.463852 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0809 01:57:19.463857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 01:57:19.463870 (+ 13us) server_negotiation.cc:407] Connection header received
0809 01:57:19.463961 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 01:57:19.463965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 01:57:19.464130 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 01:57:19.464257 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 01:57:19.464495 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.465076 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.466462 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 01:57:19.466776 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 01:57:19.466888 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 01:57:19.467576 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 01:57:19.467926 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 01:57:19.468034 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 01:57:19.468114 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 01:57:19.468169 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":79,"threads_started":1}
I20260809 01:57:19.471346 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4756
I20260809 01:57:19.485126 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4861
I20260809 01:57:19.496141 3952 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIuGTYa/build/debug/bin/kudu with pid 4962
2026-08-09T01:57:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3312 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6668 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (301290 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST